Artificial Intelligence in Breast Screening - Local Validation Essential

Abstract

Abstract Artificial intelligence (AI) tools may assist breast screening mammography programmes, but evidence gaps remain, including whether AI performance is consistent across sites and over time. This study used a three-year historical dataset (58,209 cases) from a UK regional screening programme with known clinical outcomes. The performance of a commercially available breast screening AI algorithm, used to recall women for further investigation, was evaluated. The AI algorithm was used with a pre-specified and a site-optimised threshold. The pre-specified threshold resulted in high recall rates (47.7%) which reduced to 13.0% following threshold optimisation, closer to the observed service level (5.0%). Stand-alone, the AI algorithm would have recalled 277/303 (91.4%) of cancers detected through the routine screening programme and 14/52 (26.9%) of cancers diagnosed between screening cycles (interval cancers). An approximately three-fold increase in recall rate was observed following a software upgrade on the mammography X-ray units. To ensure safe deployment, AI algorithm performance and decision thresholds should be validated when applied in new clinical settings. Real-time quality assurance systems will need to be in place to monitor AI performance in a clinical setting. Collectively these findings show that the generalisability of a breast screening AI algorithm is not guaranteed.