Author:
Davies Nathan P,Wilson Robert,Winder Madeleine S,Tunster Simon J,McVicar Kathryn,Thakrar Shivan,Williams Joe,Reid Allan
Abstract
Abstract
Background
Artificial intelligence-based large language models, like ChatGPT, have been rapidly assessed for both risks and potential in health-related assessment and learning. However, their applications in public health professional exams have not yet been studied. We evaluated the performance of ChatGPT in part of the Faculty of Public Health’s Diplomat exam (DFPH).
Methods
ChatGPT was provided with a bank of 119 publicly available DFPH question parts from past papers. Its performance was assessed by two active DFPH examiners. The degree of insight and level of understanding apparently displayed by ChatGPT was also assessed.
Results
ChatGPT passed 3 of 4 papers, surpassing the current pass rate. It performed best on questions relating to research methods. Its answers had a high floor. Examiners identified ChatGPT answers with 73.6% accuracy and human answers with 28.6% accuracy. ChatGPT provided a mean of 3.6 unique insights per question and appeared to demonstrate a required level of learning on 71.4% of occasions.
Conclusions
Large language models have rapidly increasing potential as a learning tool in public health education. However, their factual fallibility and the difficulty of distinguishing their responses from that of humans pose potential threats to teaching and learning.
Publisher
Springer Science and Business Media LLC
Subject
Education,General Medicine
Reference20 articles.
1. Holzinger A, Keiblinger K, Holub P, Zatloukal K, Müller H. AI for life: Trends in artificial intelligence for biotechnology. N Biotechnol. 2023;74:16–24.
2. Introducing CGPT. https://openai.com/blog/chatgpt. Accessed 5 Jun 2023.
3. De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, Rizzo C. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023;11:1567.
4. Centre for AI Safety Statement on AI Risk. https://www.safe.ai/statement-on-ai-risk. Accessed 5 Jun 2023.
5. Kickbusch I, Allen L, Franz C. The commercial determinants of health. Lancet Glob Health. 2016;4:e895–6.