Evaluation of the Performance of an Artificial Intelligence (AI) Algorithm in Detecting Thoracic Pathologies on Chest Radiographs

Abstract

The purpose of the study was to assess the performance of readers in diagnosing thoracic anomalies on standard chest radiographs (CXRs) with and without a deep-learning-based AI tool (Rayvolve) and to evaluate the standalone performance of Rayvolve in detecting thoracic pathologies on CXRs. This retrospective multicentric study was conducted in two phases. In phase 1, nine readers independently reviewed 900 CXRs from imaging group A and identified thoracic abnormalities with and without AI assistance. A consensus from three radiologists served as the ground truth. In phase 2, the standalone performance of Rayvolve was evaluated on 1500 CXRs from imaging group B. The average values of AUC across the readers significantly increased by 15.94%, with AI-assisted reading compared to unaided reading (0.88 ± 0.01 vs. 0.759 ± 0.07, p < 0.001). The time taken to read the CXRs decreased significantly, by 35.81% with AI assistance. The average values of sensitivity and specificity across the readers increased significantly by 11.44% and 2.95% with AI-assisted reading compared to unaided reading (0.857 ± 0.02 vs. 0.769 ± 0.02 and 0.974 ± 0.01 vs. 0.946 ± 0.01, p < 0.001). From the standalone perspective, the AI model achieved an average sensitivity, specificity, PPV, and NPV of 0.964, 0.844, 0.757, and 0.9798. The speed and performance of the readers improved significantly with AI assistance.