Comparing the quality of ChatGPT- and physician-generated responses to patients’ dermatology questions in the electronic medical record

Author:

Reynolds Kelly1,Nadelman Daniel1,Durgin Joseph1,Ansah-Addo Stephen1,Cole Daniel1,Fayne Rachel1,Harrell Jane1,Ratycz Madison1,Runge Mason1,Shepard-Hayes Amanda1,Wenzel Daniel1,Tejasvi Trilokraj1

Affiliation:

1. Department of Dermatology, University of Michigan , Ann Arbor, MI , USA

Abstract

Abstract Background ChatGPT is a free artificial intelligence (AI)-based natural language processing tool that generates complex responses to inputs from users. Objectives To determine whether ChatGPT is able to generate high-quality responses to patient-submitted questions in the patient portal. Methods Patient-submitted questions and the corresponding responses from their dermatology physician were extracted from the electronic medical record for analysis. The questions were input into ChatGPT (version 3.5) and the outputs extracted for analysis, with manual removal of verbiage pertaining to ChatGPT’s inability to provide medical advice. Ten blinded reviewers (seven physicians and three nonphysicians) rated and selected their preference in terms of ‘overall quality’, ‘readability’, ‘accuracy’, ‘thoroughness’ and ‘level of empathy’ of the physician- and ChatGPT-generated responses. Results Thirty-one messages and responses were analysed. Physician-generated responses were vastly preferred over the ChatGPT ­responses by the physician and nonphysician reviewers and received significantly higher ratings for ‘readability’ and ‘level of empathy’. Conclusions The results of this study suggest that physician-generated responses to patients’ portal messages are still preferred over ChatGPT, but generative AI tools may be helpful in generating the first drafts of responses and providing information on education resources for patients.

Publisher

Oxford University Press (OUP)

Cited by 4 articles. 订阅此论文施引文献 订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献

同舟云学术

1.学者识别学者识别

2.学术分析学术分析

3.人才评估人才评估

"同舟云学术"是以全球学者为主线,采集、加工和组织学术论文而形成的新型学术文献查询和分析系统,可以对全球学者进行文献检索和人才价值评估。用户可以通过关注某些学科领域的顶尖人物而持续追踪该领域的学科进展和研究前沿。经过近期的数据扩容,当前同舟云学术共收录了国内外主流学术期刊6万余种,收集的期刊论文及会议论文总量共计约1.5亿篇,并以每天添加12000余篇中外论文的速度递增。我们也可以为用户提供个性化、定制化的学者数据。欢迎来电咨询!咨询电话:010-8811{复制后删除}0370

www.globalauthorid.com

TOP

Copyright © 2019-2024 北京同舟云网络信息技术有限公司
京公网安备11010802033243号  京ICP备18003416号-3