A comparative vignette study: Evaluating the potential role of a generative AI model in enhancing clinical decision‐making in nursing

Abstract

AbstractAimThis study explores the potential of a generative artificial intelligence tool (ChatGPT) as clinical support for nurses. Specifically, we aim to assess whether ChatGPT can demonstrate clinical decision‐making equivalent to that of expert nurses and novice nursing students. This will be evaluated by comparing ChatGPT responses to clinical scenarios to those of nurses on different levels of experience.DesignThis is a cross‐sectional study.MethodsEmergency room registered nurses (i.e. experts; n = 30) and nursing students (i.e. novices; n = 38) were recruited during March–April 2023. Clinical decision‐making was measured using three validated clinical scenarios involving an initial assessment and reevaluation. Clinical decision‐making aspects assessed were the accuracy of initial assessments, the appropriateness of recommended tests and resource use and the capacity to reevaluate decisions. Performance was also compared by timing response generations and word counts. Expert nurses and novice students completed online questionnaires (via Qualtrics), while ChatGPT responses were obtained from OpenAI.ResultsConcerning aspects of clinical decision‐making and compared to novices and experts: (1) ChatGPT exhibited indecisiveness in initial assessments; (2) ChatGPT tended to suggest unnecessary diagnostic tests; (3) When new information required re‐evaluation, ChatGPT responses demonstrated inaccurate understanding and inappropriate modifications. In terms of performance, the mean number of words utilized in ChatGPT answers was 27–41 times greater than that utilized by both experts and novices; and responses were provided approximately 4 times faster than those of novices and twice faster than expert nurses. ChatGPT responses maintained logical structure and clarity.ConclusionsA generative AI tool demonstrated indecisiveness and a tendency towards over‐triage compared to human clinicians.ImpactThe study shows that it is important to approach the implementation of ChatGPT as a nurse's digital assistant with caution. More study is needed to optimize the model's training and algorithms to provide accurate healthcare support that aids clinical decision‐making.Reporting methodThis study adhered to relevant EQUATOR guidelines for reporting observational studies.Patient or public contributionPatients were not directly involved in the conduct of this study.