Evaluating GPT-4-based ChatGPT’s Clinical Potential on the NEJM Quiz-Reference-Cited by-同舟云学术

Evaluating GPT-4-based ChatGPT’s Clinical Potential on the NEJM Quiz

Published:2023-05-05 Issue: Volume: Page:
ISSN:
Container-title:
language:
Short-container-title:

Author:

Ueda Daiju^ORCID,Walston Shannon L,Matsumoto Toshimasa,Deguchi Ryo,Tatekawa Hiroyuki,Miki Yukio

Abstract

AbstractBackgroundGPT-4-based ChatGPT demonstrates significant potential in various industries; however, its potential clinical applications remain largely unexplored.MethodsWe employed the New England Journal of Medicine (NEJM) quiz “Image Challenge” from October 2021 to March 2023 to assess ChatGPT’s clinical capabilities. The quiz, designed for healthcare professionals, tests the ability to analyze clinical scenarios and make appropriate decisions. We evaluated ChatGPT’s performance on the NEJM quiz, analyzing its accuracy rate by questioning type and specialty after excluding quizzes which were impossible to answer without images. The NEJM quiz has five multiple-choice options, but ChatGPT was first asked to answer without choices, and then given the choices to answer afterwards, in order to evaluate the accuracy in both scenarios.ResultsChatGPT achieved an 87% accuracy without choices and a 97% accuracy with choices, after excluding 16 image-based quizzes. Upon analyzing performance by quiz type, ChatGPT excelled in the Diagnosis category, attaining 89% accuracy without choices and 98% with choices. Although other categories featured fewer cases, ChatGPT’s performance remained consistent. It demonstrated strong performance across the majority of medical specialties; however, Genetics had the lowest accuracy at 67%.ConclusionChatGPT demonstrates potential for clinical application, suggesting its usefulness in supporting healthcare professionals and enhancing AI-driven healthcare.

Publisher

Cold Spring Harbor Laboratory

Reference16 articles.

1. Advances in natural language processing

2. OpenAI. GPT-4 Technical Report [Internet]. arXiv [cs.CL]. 2023;Available from: http://arxiv.org/abs/2303.08774

3. Brown TB , Mann B , Ryder N , et al. Language Models are Few-Shot Learners [Internet]. arXiv [cs.CL]. 2020 [cited 2023 Apr 8];1877–901. Available from: https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html

4. Eloundou T , Manning S , Mishkin P , Rock D. GPTs are GPTs: An early look at the labor market impact potential of large language models [Internet]. arXiv [econ.GN]. 2023;Available from: http://arxiv.org/abs/2303.10130

5. Bubeck S , Chandrasekaran V , Eldan R , et al. Sparks of Artificial General Intelligence: Early experiments with GPT-4 [Internet]. arXiv [cs.CL]. 2023;Available from: http://arxiv.org/abs/2303.12712

Cited by 8 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Les Printemps de la Médecine Interne : l’intelligence artificielle face aux experts internistes;La Revue de Médecine Interne;2024-07

2. Labeling Radiology Report With GPT-4 Prompt Engineering: Comparative Study of in-Context Prompting (Preprint);2024-03-15

3. Diagnostic Performance Comparison between Generative AI and Physicians: A Systematic Review and Meta-Analysis;2024-01-22

4. Assessing the research landscape and clinical utility of large language models: A scoping review;2023-10-30

5. Comparative Analysis of ChatGPT’s Diagnostic Performance with Radiologists Using Real-World Radiology Reports of Brain Tumors;2023-10-28