The critical need for expert oversight of ChatGPT: Prompt engineering for safeguarding child healthcare information

Abstract

Abstract Objective ChatGPT and other large language models have the potential to transform the health information landscape online. However, lack of domain-specific expertise and known errors in large language models raise concerns about the widespread adoption of content generated by these tools for parents making healthcare decisions for their children. The aim of this study is to determine if health-related text generated by ChatGPT under the supervision of an expert is comparable to that generated by an expert regarding persuasiveness and credibility from the perspective of a parent. Methods In a cross-sectional study 116 parents aged 18–65 years (M = 45.02, SD = 10.92) were asked to complete a baseline assessment of their behavioral intentions regarding pediatric healthcare topics. Subsequently, participants were asked to rate text generated by either an expert or by ChatGPT under supervision of an expert. Results Results indicate that prompt engineered ChatGPT is capable of impacting behavioral intentions for medication, sleep, and diet decision-making. Additionally, there was little distinction between prompt engineered ChatGPT and content experts on perceived morality, trustworthiness, expertise, accuracy, and reliance. Notably, when differences were present, prompt engineered ChatGPT was rated as higher in trustworthiness and accuracy, and participants indicated they would be more likely to rely on the information presented by prompt engineered ChatGPT compared to the expert. Discussion Given that parents will trust and rely on information generated by ChatGPT, it is critically important that human domain-specific expertise be applied to healthcare information that will ultimately be presented to consumers (e.g., parents).