Boosting LLM-Assisted Diagnosis: 10-Minute LLM Tutorial Elevates Radiology Residents’ Performance in Brain MRI Interpretation

Abstract

AbstractPurposeTo evaluate the impact of a structured tutorial on the use of a large language model (LLM)-based search engine on radiology residents’ performance in LLM-assisted brain MRI differential diagnosis.Materials & MethodsIn this retrospective study, nine radiology residents determined the three most likely differential diagnoses for three sets of ten brain MRI cases with a challenging yet definite diagnosis. Each set of cases was assessed 1) with the support of conventional internet search, 2) using an LLM-based search engine (© Perplexity AI) without prior training, or 3) with LLM assistance after a structured 10-minute tutorial on how to effectively use the tool for differential diagnosis. The tutorial content was based on the results of two studies on LLM-assisted radiological diagnosis and included a prompt template. Reader responses were rated using a binary and numeric scoring system. Reading times were tracked and confidence levels were recorded on a 5-point Likert scale. Binary and numeric scores were analyzed using chi-square tests and pairwise Mann-Whitney U tests each. Search engine logs were examined to quantify user interaction metrics, and to identify hallucinations and misinterpretations in LLM responses.ResultsRadiology residents achieved the highest accuracy when employing the LLM-based search engine following the tutorial, indicating the correct diagnosis among the top three differential diagnoses in 62.5% of cases (55/88). This was followed by the LLM-assisted workflow before the tutorial (44.8%; 39/87) and the conventional internet search workflow (32.2%; 28/87). The LLM tutorial led to significantly higher performance (binary scores: p = 0.042, numeric scores: p = 0.016) and confidence (p = 0.006) but resulted in no relevant differences in reading times. Hallucinations were found in 5.1% of LLM queries.ConclusionA structured 10-minute LLM tutorial increased performance and confidence levels in LLM-assisted brain MRI differential diagnosis among radiology residents.Clinical Relevance StatementOur findings highlight the considerable benefits that even low-cost, low-effort educational interventions on LLMs can provide. Integrating LLM education in radiology training programs could augment practitioners’ capacity to harness AI technologies effectively.