Abstract
AbstractIntroductionAI based software, including computer aided detection software for chest radiographs (CXR-CAD), was developed during the pandemic to improve COVID-19 case finding and triage. In high burden TB countries, the use of highly portable CXR and computer aided detection software has been adopted more broadly to improve the screening and triage of individuals for TB, but there is little evidence in these settings regarding COVID-19 CAD performance.MethodsWe performed a multicenter, retrospective cross-over study evaluating CXRs from individuals at risk for COVID-19. We evaluated performance of CAD software and radiologists in comparison to COVID-19 laboratory results in 671 individuals evaluated for COVID-19 at sites in Zambia and Malawi between January 2021 and June 2022. All CXRs were interpreted by an expert radiologist and two commercially available COVID-19 CXR-CAD software.ResultsRadiologists interpreted CXRs for COVID-19 with a sensitivity of 73% (95% CI: 69%-76%) and specificity of 49% (95% CI: 40%-58%). One CAD software (CAD2) showed performance in diagnosing COVID-19 that was comparable to that of radiologists, (AUC-ROC of 0.70 (95% CI: 0.65-0.75)), while a second (CAD1) showed inferior performance (AUC-ROC of 0.57 (95% CI: 0.52-0.63)). Agreement between CAD software and radiologists was moderate for diagnosing COVID-19, and very good agreement in differentiating normal and abnormal CXRs in this high prevalent population.ConclusionsThe study highlights the potential of CXR-CAD as a tool to support effective triage of individuals in Malawi and Zambia during the pandemic, particularly for distinguishing normal from abnormal CXRs. These findings suggest that while current AI-based diagnostics like CXR-CAD show promise, their effectiveness varies significantly. In order to better prepare for future pandemics, there is a need for representative training data to optimize performance in key populations, and ongoing data collection to maintain diagnostic accuracy, especially as new disease strains emerge.Author SummaryDuring the COVID-19 pandemic, AI-based software was developed to help identify and manage cases, including software that assists in reading chest X-rays (CXR-CAD). This technology has also been used in high tuberculosis (TB) burden countries to screen and manage TB cases. However, there’s limited information on how well these tools work for COVID-19 in these settings. This study examined chest X-rays from people at risk for COVID-19 in Zambia and Malawi to evaluate the performance of CXR-CAD software against expert radiologists and laboratory COVID-19 tests. The research included X-rays from 671 participants, reviewed by two AI software programs and radiologists.The results showed that radiologists had a sensitivity of 73% and specificity of 49% in detecting COVID-19. One AI software (CAD2) performed similarly to radiologists, while another (CAD1) performed worse. The agreement between the AI software and radiologists varied, but both were good at distinguishing between normal and abnormal X-rays.The study suggests that while AI tools like CXR-CAD show potential, their effectiveness can vary. To improve these tools for future pandemics, more representative training data and continuous data collection are necessary.
Publisher
Cold Spring Harbor Laboratory