BACKGROUND
Epilepsy is a neurological disorder, and accurate information about the condition is crucial for those affected. As natural language processing models such as ChatGPT become increasingly popular for accessing medical information, it is important to evaluate their accuracy and reliability.
OBJECTIVE
This study aimed to investigate the accuracy of ChatGPT in providing educational information related to epilepsy.
METHODS
We evaluated the accuracy and educational quality of ChatGPT's responses to 57 questions commonly asked by epilepsy patients or their caregivers using the “Epilepsy Patient and Caregiver Guide” from the Korean Epilepsy Society. The questions were divided into categories and entered into ChatGPT 4 version and 3.5 for comparison. Responses were evaluated by two epileptologists, with a third epileptologist serving as the final arbiter in cases of disagreement. ChatGPT 4's performance was compared to that of ChatGPT 3.5 and the official guide. Evaluations were categorized into four grades and compared using a standardized method in cases of disagreement.
RESULTS
Among the 57 questions, 40 responses had “sufficient educational value,” while 16 were “correct but inadequate.” Only one response was “mixed with correct and incorrect” information, and no responses were found to be “incorrect.” Comparison with ChatGPT 3.5 showed that GPT-4 performed better or similar in most questions, except for two instances where GPT-3.5 provided better answers. Compared to the official epilepsy guide, GPT-4 also had better or similar responses for most of the questions, with four exceptions where the official guide had better answers.
CONCLUSIONS
This study demonstrates the potential of ChatGPT-4 as a tool for providing accessible and helpful medical information related to epilepsy. Although there are many concerns about the use of GPT models for medical education, they can still be considered to decrease the burden on healthcare professionals with respect to patient education. More research is needed to fully understand the potential benefits and limitations of using language models such as ChatGPT in healthcare and medicine.