Can generative artificial intelligence provide accurate medical advice?: a case of ChatGPT versus Congress of Neurological Surgeons management of acute cervical spine and spinal cord injuries clinical guidelines

Abstract

Study Design: An experimental study.Purpose: To explore the concordance of ChatGPT responses with established national guidelines for the management of cervical spine and spinal cord injuries.Overview of Literature: ChatGPT-4.0 is an artificial intelligence model that can synthesize large volumes of data and may provide surgeons with recommendations for the management of spinal cord injuries. However, no available literature has quantified ChatGPT’s capacity to provide accurate recommendations for the management of cervical spine and spinal cord injuries.Methods: Referencing the “Management of acute cervical spine and spinal cord injuries” guidelines published by the Congress of Neurological Surgeons (CNS), a total of 36 questions were formulated. Questions were stratified into therapeutic, diagnostic, or clinical assessment categories as seen in the guidelines. Questions were secondarily grouped according to whether the corresponding recommendation contained level I evidence (highest quality) versus only level II/III evidence (moderate and low quality). ChatGPT-4.0 was prompted with each question, and its responses were assessed by two independent reviewers as “concordant” or “nonconcordant” with the CNS clinical guidelines. “Nonconcordant” responses were rationalized into “insufficient” and “contradictory” categories.Results: In this study, 22/36 (61.1%) of ChatGPT’s responses were concordant with the CNS guidelines. ChatGPT’s responses aligned with 17/24 (70.8%) therapeutic questions and 4/7 (57.1%) diagnostic questions. ChatGPT’s response aligned with only one of the five clinical assessment questions. Notably, the recommendations supported by level I evidence were the least likely to be replicated by ChatGPT. ChatGPT’s responses agreed with 80.8% of the recommendations supported exclusively by level II/III evidence.Conclusions: ChatGPT-4 was moderately accurate when generating recommendations that aligned with the clinical guidelines. The model frequently aligned with low evidence and therapeutic recommendations but exhibited inferior performance on topics that contained high-quality evidence or pertained to diagnostic and clinical assessment strategies. Medical practitioners should monitor its usage until further models can be rigorously trained on medical data.