Performance and limitation of machine learning algorithms for diabetic retinopathy screening: A meta-analysis (Preprint)

Abstract

Standardly diagnosed by human experts, the high prevalence of diabetic retinopathy (DR) warrants a more efficient screening method. Although machine learning (ML)-based automated DR diagnosis has gained attention due to recent approval of IDx-DR, performance of this tool has not be examined systematically, and the best ML technique for utilization in real-world setting has not been discussed.

To examine systematically the overall diagnostic accuracy of ML in diagnosing DR of different categories based on color fundus photographs and to determine the state-of-the-art ML approach.

Published studies in PubMed and EMBASE were searched from inception to June, 2020. Studies were screened for relevant outcomes, publication types, and data sufficiency, and a total of 60 (2.8%) out of 2128 studies were retrieved after study selection. Extraction of data was performed by 2 authors according to PRISMA, and the quality assessment was performed according to QUADUS-2. Meta-analysis of diagnostic accuracy was pooled using a bivariate random-effects model. The main outcomes included diagnostic accuracy, sensitivity, and specificity of ML in diagnosing DR based on color fundus photographs, as well as the performances of different major types of ML algorithms.

The primary meta-analysis included 60 color fundus photograph studies (445,175 interpretations). Overall, ML demonstrated high accuracy in diagnosing DR of various categories, with a pooled AUROC from 0.97 (95% CI: 0.96, 0.99) to 0.99 (95%CI: 0.98, 1.00). The performance of ML in detecting more-than-mild DR (mtmDR) was robust (Sen: 0.95, AUROC: 0.97), and by subgroup analyses, we observed that robust performance of ML was not limited to benchmark datasets (Sen: 0.92; AUROC: 0.96) but could be generalized to images collected in clinical practice (Sen: 0.97; AUROC: 097). Neural network was the most widely utilized method, and the subgroup analysis revealed a pooled AUROC of 0.98 (95% CI: 0.96, 0.99) for studies that utilized neural networks to diagnose mtmDR.

This meta-analysis demonstrated high diagnostic accuracy of ML algorithms in detecting diabetic retinopathy on color fundus photographs, suggesting that state-of-the-art, ML-based DR screening algorithms are likely ready for clinical applications. However, a significant portion of the earlier published studies had methodology flaws, such as the lack of external validation and presence of spectrum bias. The results of these studies should be interpreted with caution.