Performance and Consistency of ChatGPT‐4 Versus Otolaryngologists: A Clinical Case Series

Abstract

AbstractObjectiveTo study the performance of Chatbot Generative Pretrained Transformer‐4 (ChatGPT‐4) in the management of cases in otolaryngology–head and neck surgery.Study DesignProspective case series.SettingMulticenter University Hospitals.MethodsHistory, clinical, physical, and additional examinations of adult outpatients consulting in otolaryngology departments of CHU Saint‐Pierre and Dour Medical Center were presented to ChatGPT‐4, which was interrogated for differential diagnoses, management, and treatment(s). According to specialty, the ChatGPT‐4 responses were assessed by 2 distinct, blinded board‐certified otolaryngologists with the Artificial Intelligence Performance Instrument.ResultsOne hundred cases were presented to ChatGPT‐4. ChaGPT‐4 indicated a mean of 3.34 (95% confidence interval [CI]: 3.09, 3.59) additional examinations per patient versus 2.10 (95% CI: 1.76, 2.34; P = .001) for the practitioners. There was strong consistency (k > 0.600) between otolaryngologists and ChatGPT‐4 for the indication of upper aerodigestive tract endoscopy, positron emission tomography and computed tomography, audiometry, tympanometry, and psychophysical evaluations. Primary diagnosis was correctly performed by ChatGPT‐4 in 38% to 86% of cases depending on subspecialty. Additional examinations indicated by ChatGPT‐4 were pertinent and necessary in 8% to 31% of cases, while the treatment regimen was pertinent in 12% to 44% of cases. The performance of ChatGPT‐4 was not influenced by the human‐reported level of difficulty of clinical cases.ConclusionChatGPT‐4 may be a promising adjunctive tool in otolaryngology, providing extensive documentation about additional examinations, primary and differential diagnoses, and treatments. The ChatGPT‐4 is more effective in providing a primary diagnosis, and less effective in the selection of additional examinations and treatments.