Performance of GPT-4 and mainstream Chinese Large Language Models on the Chinese Postgraduate Examination dataset: Potential for AI-assisted Traditional Chinese Medicine

Abstract

ChatGPT is a well-known example of a Large Language Models(LLMs) that has performed notably well in the medical licensing exams of many countries. Tradition Chinese Medicine(TCM) has gained increasing attention and concern from the international medical community. In China, the medical master’s degree exam carries more weight than the licensing exam due to the multi-level doctor training system. However, the current study lacks an evaluation of the performance of various LLMs in TCM postgraduate examinations. Consequently, we created a test dataset of LLMs using postgraduate examination questions to comprehensively evaluate the performance of GPT-4 and mainstream Chinese LLMs in responding to knowledge and clinical inquiries about TCM. Besides computing the exam scores and presenting LLM's performance on various subjects, we evaluated the output responses based on three qualitative metrics: logical reasoning, and the ability to use internal and external information. The results indicated that Ernie Bot and ChatGLM's expertise in TCM surpassed the passing threshold for the postgraduate selection examination, showcasing their enormous potential as TCM support tools.