Affiliation:
1. Department of Computer Science Aalborg University Copenhagen Denmark
2. Department of Dermatology Copenhagen University Hospital, Bispebjerg and Frederiksberg Copenhagen Denmark
Abstract
AbstractBackgroundThe potential applications of artificial intelligence (AI) in dermatology are evolving rapidly. Chatbots are an emerging trend in healthcare that rely on large language models (LLMs) to generate answers to prompts from users. However, the factuality and user experience (UX) of such chatbots remain to be evaluated in the context of dermato‐oncology.ObjectivesTo examine the potential of Chat Generative Pretrained Transformer (ChatGPT) as a reliable source of information in the context of actinic keratosis (AK) and to evaluate clinicians' attitudes and UX with regard to the chatbot.MethodsA set of 38 clinical questions were compiled and entered as natural language queries in separate, individual conversation threads in ChatGPT (OpenAI, default GPT 3.5). Questions pertain to patient education, diagnosis, and treatment. ChatGPT's responses were presented to a panel of 7 dermatologists for rating of factual accuracy, currency of information, and completeness of the response. Attitudes towards ChatGTP were explored qualitatively and quantitatively using a validated user experience questionnaire (UEQ).ResultsChatGPT answered 12 questions (31.6%) with an accurate, current, and complete response. ChatGPT performed best for questions on patient education, including pathogenesis of AK and potential risk factors, but struggled with diagnosis and treatment. Major deficits were seen in grading AK, providing up‐to‐date treatment guidance, and asserting incorrect information with unwarranted confidence. Further, responses were considered verbose with an average word count of 198 (SD 55) and overly alarming of the risk of malignant transformation. Based on UEQ responses, the expert panel considered ChatGPT an attractive and efficient tool, scoring highest for speed of information retrieval, but deemed the chatbot inaccurate and verbose, scoring lowest for clarity.ConclusionsWhile dermatologists rated ChatGPT high in UX, the underlying LLMs that enable such chatbots require further development to guarantee accuracy and concision required in a clinical setting.
Reference24 articles.
1. The natural history of actinic keratosis: a systematic review
2. ChatGPT and antimicrobial advice: the end of the consulting infection doctor?
3. DevlinJ ChangMW LeeK ToutanovaK. BERT: pre‐training of Deep Bidirectional Transformers for Language Understanding.Proc 2019 Conf North.2019;4171–4186.https://aclanthology.org/N19-1423/
4. ScaoTL FanA AkikiC PavlickE IlićS HesslowD et al. BLOOM: a 176B‐parameter open‐access multilingual language model. 2022 [cited 2023 Mar 15]. Available from:https://arxiv.org/abs/2211.05100
5. The Lancet Digital Health. ChatGPT: friend or foe? Lancet Digit Health.2023;5(3):e102.