Simulating Success: GPT-4’s Impact on the Development of a Virtual Communication Training Skills Module for Medical Students (Preprint)

Abstract

Standardized patients (SPs) prepare medical students (MSs) for difficult conversations with patients, such as discussions about life-changing diagnostic results. Despite their value, SP training is constrained by available resources and competing clinical demands. Researchers are turning to generative pre-trained transformers (GPTs) and other large language models (LLMs) to create communication skills simulations that incorporate computer-generated (virtual) SPs (VSPs). GPT-4 is a major LLM advance that makes it practical for developers to use text-based prompts instead of Branching Path Simulations (BPS) that rely on pre-scripted conversations. These nascent developmental practices have yet to take root in the literature to guide other researchers in developing their own simulations.

This study aims to describe our developmental process and lessons learned for a GPT-4-driven VSP. We designed the VSP to help MS learners rehearse discussing abnormal mammography results with a patient as a primary care physician (PCP).

We conducted in-depth interviews with 5 MSs, 5 PCPs, and 5 breast cancer survivors to inform development of the scenario and VSP. We then used Hyperskill, simulation authoring software, to develop a VSP. Initially, GPT-4 was not available. We started development using BPS. Aided by GPT-4, we used a prompt to instruct the VSP regarding the scenario, its emotional state, and expectations for how the learner should converse with it. We iteratively refined the prompt after multiple rounds of testing. As an exploratory feature, we programmed the simulation to display written feedback on the learner’s performance in communicating with the VSP.

In-depth interviews helped us create a realistic scenario by establishing when a conversation between a PCP and patient would likely first take place in the breast cancer screening process and the mode of communication. The scenario simulates a telephone call between the learner and patient to discuss the results of an abnormal diagnostic mammogram that requires a biopsy. Interviews informed programming of prompts for the VSP to expect learner communication based on the SPIKES protocol for delivering bad news. The simulation also evaluated the learner’s performance based on the SPIKES protocol. Preliminary testing was promising. The VSP asked sensible questions about their mammography results and responded to learner inquiries using a realistic voice replete with appropriate emotional inflections based on the conversation. Feedback was useful to highlight major SPIKES deviations but less so when clinical judgement was warranted to balance VSP responses with appropriate next steps (e.g., not pressuring the VSP to schedule a biopsy while displaying agitation).

GPT-4 streamlined development and provided a better and more natural user experience than what we were able to provide using BPS. As next steps, we will continue to develop the simulation to improve feedback and pilot test the VSP with MSs to evaluate its feasibility.