Evaluating the Diagnostic Performance of Large Language Models in Identifying Complex Multisystemic Syndromes: A Comparative Study with Radiology Residents

Abstract

AbstractAimThis study evaluates the diagnostic capabilities of large language models (LLMs) in interpreting imaging patterns, focusing on their utility as a resource for radiology residents. We compare the diagnostic performance of OpenAI’s GPT-3.5, GPT-4, and Google’s Gemini Pro against radiology residents in identifying complex, multisystemic syndromes with an increased risk of cancer.MethodsWe assessed diagnostic accuracy using textual descriptions of radiological findings from 60 diseases selected from The Familial Cancer Database. Participants included three LLMs and three radiology residents. Diagnostic responses were scored on accuracy and first choice correctness. Experiments with AI models were conducted using default API settings.ResultsGPT-4 achieved the highest diagnostic accuracy (63%) and first choice accuracy (40%), significantly outperforming the radiology residents whose accuracy ranged from 22% to 43%. The overall average accuracy for AI models was 49.3%, compared to 29.0% for residents. Error analysis revealed that while some diseases were universally recognized, others highlighted diagnostic challenges across both human and AI participants.ConclusionGPT-4 outperforms radiology residents in diagnosing complex, infrequent multisystemic diseases. These findings suggest potential benefits of integrating AI tools to improve diagnostic accuracy for rare conditions and imply a need for revisions in medical training to incorporate AI competencies, enhancing diagnostic processes and resident education in radiology.