A machine learning model based on readers’ characteristics to predict their performances in reading screening mammograms

Abstract

Abstract Objectives Proposing a machine learning model to predict readers’ performances, as measured by the area under the receiver operating characteristics curve (AUC) and lesion sensitivity, using the readers’ characteristics. Methods Data were collected from 905 radiologists and breast physicians who completed at least one case-set of 60 mammographic images containing 40 normal and 20 biopsy-proven cancer cases. Nine different case-sets were available. Using a questionnaire, we collected radiologists’ demographic details, such as reading volume and years of experience. These characteristics along with a case set difficulty measure were fed into two ensemble of regression trees to predict the readers’ AUCs and lesion sensitivities. We calculated the Pearson correlation coefficient between the predicted values by the model and the actual AUC and lesion sensitivity. The usefulness of the model to categorize readers as low and high performers based on different criteria was also evaluated. The performances of the models were evaluated using leave-one-out cross-validation. Results The Pearson correlation coefficient between the predicted AUC and actual one was 0.60 (p < 0.001). The model’s performance for differentiating the reader in the first and fourth quartile based on the AUC values was 0.86 (95% CI 0.83–0.89). The model reached an AUC of 0.91 (95% CI 0.88–0.93) for distinguishing the readers in the first quartile from the fourth one based on the lesion sensitivity. Conclusion A machine learning model can be used to categorize readers as high- or low-performing. Such model could be useful for screening programs for designing a targeted quality assurance and optimizing the double reading practice.