Assessing the appropriateness and completeness of ChatGPT-4’s AI-generated responses for queries related to diabetic retinopathy

Abstract

Objective: To evaluate the appropriateness of responses generated by an online chat-based artificial intelligence (AI) model for diabetic retinopathy (DR) related questions. Design: Cross-sectional study. Methods: A set of 20 questions framed from the patient’s perspective addressing DR-related queries, such as the definition of disease, symptoms, prevention methods, treatment options, diagnostic methods, visual impact, and complications, were formulated for input into ChatGPT-4. Peer-reviewed, literature-based answers were collected from popular search engines for the selected questions and three retinal experts reviewed the responses. An inter-human agreement was analyzed for consensus expert responses and also between experts. The answers generated by the AI model were compared with those provided by the experts. The experts rated the response generated by ChatGPT-4 on a scale of 0–5 for appropriateness and completeness. Results: The answers provided by ChatGPT-4 were appropriate and complete for most of the DR-related questions. The response to questions on the adverse effects of laser photocoagulation therapy and compliance to treatment was not perfectly complete. The average rating given by the three retina expert evaluators was 4.84 for appropriateness and 4.38 for completeness of answers provided by the AI model. This corresponds to an overall 96.8% agreement among the experts for appropriateness and 87.6% for completeness regarding AI-generated answers. Conclusion: ChatGPT-4 exhibits a high level of accuracy in generating appropriate responses for a range of questions in DR. However, there is a need to improvise the model to generate complete answers for certain DR-related topics.