Limitations of ChatGPT for Patient Education Regarding Frequently Searched Questions About Benign Prostatic Hyperplasia

Abstract

Purpose: Generative Pre-Trained Transformer 4 (GPT-4) or ChatGPT, an artificial intelligence language generator, has been used in multiple applications in medicine. We seek to evaluate ChatGPT’s ability to generate appropriate responses consistent with guidelines when asked patient-generated questions regarding benign prostatic hyperplasia (BPH). Methods: GPT-4 (Open AI) was queried to answer 20 of the most frequently asked questions regarding BPH according to Google’s search engine. Questions were subclassified into 5 categories: pathophysiology, diagnosis, behavioral modification, procedural interventions, and medical treatments. These responses were graded by expert urologists involved in generating the AUA BPH Clinical Guidelines on 3 domains: accuracy, comprehensiveness, and contemporaneousness. Results: Of the responses generated, 70% were accurate, 73% were comprehensive, and 63% were contemporary. Seventeen percent of GPT-4 responses were inaccurate, incomprehensive, and noncontemporary. The highest scores were in diagnosis (88.9%), pathophysiology (83.3%), and relative medical treatment (70.4%) subcategories. The most inaccurate, incomprehensive, and noncontemporary category was procedural treatments (31.7%). Conclusions: When rated by guideline experts in management of male lower urinary tract symptoms attributed to BPH, GPT-4 has been shown to provide some accurate, comprehensive, and updated information exceeding 80% accuracy, comprehensiveness and contemporaneousness in the categories of diagnosis, and pathophysiology. GPT-4 can serve as an adequate revising tool for patients to learn the basics about BPH regarding diagnosis and pathophysiology, but the current version fails patients in other subcategories such as relative medical treatments and procedural treatments. Given the rise in GPT-4’s ubiquity, it is imperative that the urologic community critically examines all sources of patient information to identify areas of improvement.