Artificial Intelligence Chatbot Performance in Triage of Ophthalmic Conditions

Abstract

AbstractImportanceAccess to human expertise for affordable and efficient triage of ophthalmic conditions is inconsistent. With recent advancements in publicly available artificial intelligence (AI) chatbots, individuals may turn to these tools for triage of ophthalmic complaints.ObjectiveTo evaluate the triage performance of AI chatbots for ophthalmic conditionsDesignCross-sectional studySettingSingle centerParticipantsOphthalmology trainees, OpenAI ChatGPT (GPT-4), Bing Chat, and WebMD Symptom CheckerMethodsForty-four clinical vignettes representing common ophthalmic complaints were developed, and a standardized pathway of prompts was presented to each tool in March 2023.Primary outcomes were proportion of responses with correct diagnosis listed in the top three possible diagnoses and proportion with correct triage urgency. Ancillary outcomes included presence of grossly inaccurate statements, mean reading grade level, mean response word count, proportion with attribution, most common sources cited, and proportion with a disclaimer regarding chatbot limitations in dispensing medical advice.ResultsThe physician respondents, ChatGPT, Bing Chat, and WebMD listed the appropriate diagnosis among the top three suggestions in 42 (95%), 41 (93%), 34 (77%), and 8 (33%) cases, respectively. Triage urgency was appropriate in 38 (86%), 43 (98%), and 37 (84%) cases for the physicians, ChatGPT, and Bing Chat, correspondingly.Conclusions and RelevanceChatGPT using the GPT-4 model offered high diagnostic and triage accuracy that was comparable to the physician respondents, with no grossly inaccurate statements. Bing Chat had lower accuracy, some instances of grossly inaccurate statements, and a tendency to overestimate triage urgency.