The effect of variable labels on deep learning models trained to predict breast density

Abstract

Abstract Purpose. High breast density is associated with reduced efficacy of mammographic screening and increased risk of developing breast cancer. Accurate and reliable automated density estimates can be used for direct risk prediction and passing density related information to further predictive models. Expert reader assessments of density show a strong relationship to cancer risk but also inter-reader variation. The effect of label variability on model performance is important when considering how to utilise automated methods for both research and clinical purposes. Methods. We utilise subsets of images with density labels from the same 13 readers and 12 reader pairs, and train a deep transfer learning model which is used to assess how label variability affects the mapping from representation to prediction. We then create two end-to-end models: one that is trained on averaged labels across the reader pairs and the second that is trained using individual reader scores, with a novel alteration to the objective function. The combination of these two end-to-end models allows us to investigate the effect of label variability on the model representation formed. Results. We show that the trained mappings from representations to labels are altered considerably by the variability of reader scores. Training on labels with distribution variation removed causes the Spearman rank correlation coefficients to rise from 0.751 ± 0.002 to either 0.815 ± 0.026 when averaging across readers or 0.844 ± 0.002 when averaging across images. However, when we train different models to investigate the representation effect we see little difference, with Spearman rank correlation coefficients of 0.846 ± 0.006 and 0.850 ± 0.006 showing no statistically significant difference in the quality of the model representation with regard to density prediction. Conclusions. We show that the mapping between representation and mammographic density prediction is significantly affected by label variability. However, the effect of the label variability on the model representation is limited.