Evaluating the Quality and Usability of Artificial Intelligence–Generated Responses to Common Patient Questions in Foot and Ankle Surgery

Abstract

Background: Artificial intelligence (AI) platforms, such as ChatGPT, have become increasingly popular outlets for the consumption and distribution of health care–related advice. Because of a lack of regulation and oversight, the reliability of health care–related responses has become a topic of controversy in the medical community. To date, no study has explored the quality of AI-derived information as it relates to common foot and ankle pathologies. This study aims to assess the quality and educational benefit of ChatGPT responses to common foot and ankle–related questions. Methods: ChatGPT was asked a series of 5 questions, including “What is the optimal treatment for ankle arthritis?” “How should I decide on ankle arthroplasty versus ankle arthrodesis?” “Do I need surgery for Jones fracture?” “How can I prevent Charcot arthropathy?” and “Do I need to see a doctor for my ankle sprain?” Five responses (1 per each question) were included after applying the exclusion criteria. The content was graded using DISCERN (a well-validated informational analysis tool) and AIRM (a self-designed tool for exercise evaluation). Results: Health care professionals graded the ChatGPT-generated responses as bottom tier 4.5% of the time, middle tier 27.3% of the time, and top tier 68.2% of the time. Conclusion: Although ChatGPT and other related AI platforms have become a popular means for medical information distribution, the educational value of the AI-generated responses related to foot and ankle pathologies was variable. With 4.5% of responses receiving a bottom-tier rating, 27.3% of responses receiving a middle-tier rating, and 68.2% of responses receiving a top-tier rating, health care professionals should be aware of the high viewership of variable-quality content easily accessible on ChatGPT. Level of Evidence: Level III, cross sectional study.