Transfer learning for the generalization of artificial intelligence in breast cancer detection: a case-control study

Abstract

Background Some researchers have questioned whether artificial intelligence (AI) systems maintain their performance when used for women from populations not considered during the development of the system. Purpose To evaluate the impact of transfer learning as a way of improving the generalization of AI systems in the detection of breast cancer. Material and Methods This retrospective case-control Finnish study involved 191 women diagnosed with breast cancer and 191 matched healthy controls. We selected a state-of-the-art AI system for breast cancer detection trained using a large US dataset. The selected baseline system was evaluated in two experimental settings. First, we examined our private Finnish sample as an independent test set that had not been considered in the development of the system (unseen population). Second, the baseline system was retrained to attempt to improve its performance in the unseen population by means of transfer learning. To analyze performance, we used areas under the receiver operating characteristic curve (AUCs) with DeLong's test. Results Two versions of the baseline system were considered: ImageOnly and Heatmaps. The ImageOnly and Heatmaps versions yielded mean AUC values of 0.82±0.008 and 0.88±0.003 in the US dataset and 0.56 (95% CI=0.50–0.62) and 0.72 (95% CI=0.67–0.77) when evaluated in the unseen population, respectively. The retrained systems achieved AUC values of 0.61 (95% CI=0.55–0.66) and 0.69 (95% CI=0.64–0.75), respectively. There was no statistical difference between the baseline system and the retrained system. Conclusion Transfer learning with a small study sample did not yield a significant improvement in the generalization of the system.