Overcoming the lack of annotated medical images by non‐expert annotation?

Abstract

AbstractObjectiveEarly diagnosis of ischemic stroke is crucial. While CT scans are recommended, interpreting them is challenging. Machine learning can aid interpretation, but the lack of large datasets hinders its development. We explored if non‐expert labeling enhances the performance of a deep learning system trained on expert‐labeled datasets.MethodsOne‐hundred eight CT datasets were labeled for suspected strokes by both an expert and a non‐expert. A DenseNet121 model was trained on either expert data alone or with added non‐expert data. Both models were tested using expert‐only data, evaluating performance metrics such as accuracy, sensitivity, and F1 score.ResultsExpert and non‐expert labeling showed strong agreement (Cohen's kappa: 0.83). Adding non‐expert data enhanced model accuracy from 86.2% to 87.7% (p < .001) and the F1 score from 86.5% to 88.4% for non‐stroke images, and from 85.7% to 87.0% for stroke images.ConclusionMachine learning performance in stroke detection can benefit from non‐expert labeled datasets. The implications for other medical conditions and the effect on larger datasets are yet to be determined. This could potentially address the issue of scarce labeled stroke imaging datasets.