Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitals-Reference-Cited by-同舟云学术

Evaluation of federated learning variations for COVID-19 diagnosis using chest radiographs from 42 US and European hospitals

Published:2022-10-10 Issue:1 Volume:30 Page:54-63
ISSN:1067-5027
Container-title:Journal of the American Medical Informatics Association
language:en
Short-container-title:

Author:

Peng Le¹,Luo Gaoxiang¹,Walker Andrew¹,Zaiman Zachary²,Jones Emma K³,Gupta Hemant⁴,Kersten Kristopher⁵,Burns John L⁶,Harle Christopher A⁷^ORCID,Magoc Tanja⁸,Shickel Benjamin⁹¹⁰,Steenburg Scott D¹¹,Loftus Tyler¹⁰¹²,Melton Genevieve B³⁴¹³¹⁴^ORCID,Gichoya Judy Wawira¹⁵,Sun Ju¹,Tignanelli Christopher J³¹³¹⁴^ORCID

Affiliation:

1. Department of Computer Science and Engineering, University of Minnesota , Minneapolis, Minnesota, USA

2. Department of Computer Science, Emory University , Atlanta, Georgia, USA

3. Department of Surgery, University of Minnesota , Minneapolis, Minnesota, USA

4. Fairview Health Services , Minneapolis, Minnesota, USA

5. Nvidia Corporation , Santa Clara, California, USA

6. The School of Medicine, Indiana University , Indianapolis, Indiana, USA

7. Department of Health Outcomes and Biomedical Informatics, University of Florida , Gainesville, Florida, USA

8. University of Florida College of Medicine , Gainesville, Florida, USA

9. Department of Medicine, University of Florida , Gainesville, Florida, USA

10. Intelligent Critical Care Center, University of Florida , Gainesville, Florida, USA

11. Department of Radiology and Imaging Sciences, Indiana University School of Medicine , Indianapolis, Indiana, USA

12. Department of Surgery, University of Florida , Gainesville, Florida, USA

13. Center for Learning Health System Sciences, University of Minnesota , Minneapolis, Minnesota, USA

14. Institute for Health Informatics, University of Minnesota , Minneapolis, Minnesota, USA

15. Department of Radiology, Emory University , Atlanta, Georgia, USA

Abstract

Abstract Objective Federated learning (FL) allows multiple distributed data holders to collaboratively learn a shared model without data sharing. However, individual health system data are heterogeneous. “Personalized” FL variations have been developed to counter data heterogeneity, but few have been evaluated using real-world healthcare data. The purpose of this study is to investigate the performance of a single-site versus a 3-client federated model using a previously described Coronavirus Disease 19 (COVID-19) diagnostic model. Additionally, to investigate the effect of system heterogeneity, we evaluate the performance of 4 FL variations. Materials and methods We leverage a FL healthcare collaborative including data from 5 international healthcare systems (US and Europe) encompassing 42 hospitals. We implemented a COVID-19 computer vision diagnosis system using the Federated Averaging (FedAvg) algorithm implemented on Clara Train SDK 4.0. To study the effect of data heterogeneity, training data was pooled from 3 systems locally and federation was simulated. We compared a centralized/pooled model, versus FedAvg, and 3 personalized FL variations (FedProx, FedBN, and FedAMP). Results We observed comparable model performance with respect to internal validation (local model: AUROC 0.94 vs FedAvg: 0.95, P = .5) and improved model generalizability with the FedAvg model (P < .05). When investigating the effects of model heterogeneity, we observed poor performance with FedAvg on internal validation as compared to personalized FL algorithms. FedAvg did have improved generalizability compared to personalized FL algorithms. On average, FedBN had the best rank performance on internal and external validation. Conclusion FedAvg can significantly improve the generalization of the model compared to other personalization FL algorithms; however, at the cost of poor internal validity. Personalized FL may offer an opportunity to develop both internal and externally validated algorithms.

Funder

Agency for Healthcare Research and Quality

Patient-Centered Outcomes Research Institute

National Institutes of Health’s

National Center for Advancing Translational Sciences

The University of Minnesota Office of the Vice President of Research

National Institute of Biomedical Imaging and Bioengineering

MIDRC

National Science Foundation

Division of Electrical, Communication & Cyber Systems

Publisher

Oxford University Press (OUP)