Will code one day run a code? Performance of language models on ACEM primary examinations and implications

Abstract

AbstractObjectiveLarge language models (LLMs) have demonstrated mixed results in their ability to pass various specialist medical examination and their performance within the field of emergency medicine remains unknown.MethodsWe explored the performance of three prevalent LLMs (OpenAI's GPT series, Google's Bard, and Microsoft's Bing Chat) on a practice ACEM primary examination.ResultsAll LLMs achieved a passing score, with scores with GPT 4.0 outperforming the average candidate.ConclusionLarge language models, by passing the ACEM primary examination, show potential as tools for medical education and practice. However, limitations exist and are discussed.