From GPT-3.5 to GPT-4.o: A Leap in AI’s Medical Exam Performance

Abstract

ChatGPT is a large language model trained on increasingly large datasets to perform diverse language-based tasks. It is capable of answering multiple-choice questions, such as those posed by diverse medical examinations. ChatGPT has been generating considerable attention in both academic and non-academic domains in recent months. In this study, we aimed to assess GPT’s performance on anatomical multiple-choice questions retrieved from medical licensing examinations in Germany. Two different versions were compared. GPT-3.5 demonstrated moderate accuracy, correctly answering 60–64% of questions from the autumn 2022 and spring 2021 exams. In contrast, GPT-4.o showed significant improvement, achieving 93% accuracy on the autumn 2022 exam and 100% on the spring 2021 exam. When tested on 30 unique questions not available online, GPT-4.o maintained a 96% accuracy rate. Furthermore, GPT-4.o consistently outperformed medical students across six state exams, with a statistically significant mean score of 95.54% compared with the students’ 72.15%. The study demonstrates that GPT-4.o outperforms both its predecessor, GPT-3.5, and a cohort of medical students, indicating its potential as a powerful tool in medical education and assessment. This improvement highlights the rapid evolution of LLMs and suggests that AI could play an increasingly important role in supporting and enhancing medical training, potentially offering supplementary resources for students and professionals. However, further research is needed to assess the limitations and practical applications of such AI systems in real-world medical practice.