Evaluation of ChatGPT pathology knowledge using board-style questions

Abstract

Abstract Objectives ChatGPT is an artificial intelligence chatbot developed by OpenAI. Its extensive knowledge and unique interactive capabilities enable its use in various innovative ways in the medical field, such as writing clinical notes and simplifying radiology reports. Through this study, we aimed to analyze the pathology knowledge of ChatGPT to advocate its role in transforming pathology education. Methods The American Society for Clinical Pathology Resident Question Bank 2022 was used to test ChatGPT, version 4. Practice tests were created in each subcategory and answered based on the input that ChatGPT provided. Questions that required interpretation of images were excluded. We analyzed ChatGPT performance and compared it with average peer performance. Results The overall performance of ChatGPT was 56.98%, lower than that of the average peer performance of 62.81%. ChatGPT performed better on clinical pathology (60.42%) than on anatomic pathology (54.94%). Furthermore, its performance was better on easy questions (68.47%) than on intermediate (52.88%) and difficult questions (37.21%). Conclusions ChatGPT has the potential to be a valuable resource in pathology education if trained on a larger, specialized medical data set. Those relying on it (in its current form) solely for the purpose of pathology training should be cautious.