Using AI-generated suggestions from ChatGPT to optimize clinical decision support-Reference-Cited by-同舟云学术

Using AI-generated suggestions from ChatGPT to optimize clinical decision support

Published:2023-04-22 Issue:7 Volume:30 Page:1237-1245
ISSN:1067-5027
Container-title:Journal of the American Medical Informatics Association
language:en
Short-container-title:

Author:

Liu Siru¹,Wright Aileen P¹²,Patterson Barron L³,Wanderer Jonathan P¹⁴,Turer Robert W⁵⁶^ORCID,Nelson Scott D¹^ORCID,McCoy Allison B¹^ORCID,Sittig Dean F⁷^ORCID,Wright Adam¹^ORCID

Affiliation:

1. Department of Biomedical Informatics, Vanderbilt University Medical Center , Nashville, Tennessee, USA

2. Department of Medicine, Vanderbilt University Medical Center , Nashville, Tennessee, USA

3. Department of Pediatrics, Vanderbilt University Medical Center , Nashville, Tennessee, USA

4. Department of Anesthesiology, Vanderbilt University Medical Center , Nashville, Tennessee, USA

5. Department of Emergency Medicine, University of Texas Southwestern Medical Center , Dallas, Texas, USA

6. Clinical Informatics Center, University of Texas Southwestern Medical Center , Dallas, Texas, USA

7. School of Biomedical Informatics, University of Texas Health Science Center , Houston, Texas, USA

Abstract

Abstract Objective To determine if ChatGPT can generate useful suggestions for improving clinical decision support (CDS) logic and to assess noninferiority compared to human-generated suggestions. Methods We supplied summaries of CDS logic to ChatGPT, an artificial intelligence (AI) tool for question answering that uses a large language model, and asked it to generate suggestions. We asked human clinician reviewers to review the AI-generated suggestions as well as human-generated suggestions for improving the same CDS alerts, and rate the suggestions for their usefulness, acceptance, relevance, understanding, workflow, bias, inversion, and redundancy. Results Five clinicians analyzed 36 AI-generated suggestions and 29 human-generated suggestions for 7 alerts. Of the 20 suggestions that scored highest in the survey, 9 were generated by ChatGPT. The suggestions generated by AI were found to offer unique perspectives and were evaluated as highly understandable and relevant, with moderate usefulness, low acceptance, bias, inversion, redundancy. Conclusion AI-generated suggestions could be an important complementary part of optimizing CDS alerts, can identify potential improvements to alert logic and support their implementation, and may even be able to assist experts in formulating their own suggestions for CDS improvement. ChatGPT shows great potential for using large language models and reinforcement learning from human feedback to improve CDS alert logic and potentially other medical areas involving complex, clinical logic, a key step in the development of an advanced learning health system.

Funder

NIH

Publisher

Oxford University Press (OUP)

Subject

Health Informatics

Link

https://academic.oup.com/jamia/article-pdf/30/7/1237/50634330/ocad072.pdf

Reference38 articles.

1. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success;Kawamoto;BMJ,2005

2. Quantifying the competitiveness of the electronic health record market and its implications for interoperability;Sorace;Int J Med Inform,2020

3. Effect of clinical decision-support systems;Bright;Ann Intern Med,2012