Benchmarking Large Language Models in Adolescent Growth and Development: A Comparative Analysis of Claude2, ChatGPT-3.5, and Google Bard

Abstract

Abstract Background: Significant attention has been drawn to large-scale language models (LLMs) for their ability to generate responses that are both contextually relevant and reminiscent of human conversation. Yet, the precision of these models in specialized medical fields, particularly those pertaining to adolescent health, remains largely unexamined. Online searches for information about common health issues during adolescent developmental stages are frequent among patients and their families. In this context, our research evaluates how effectively three different LLMs - Claude2, ChatGPT-3.5, and Google Bard - handle typical inquiries concerning adolescent growth and health development. Methods: Our research involved gathering 100 frequently asked questions about adolescent growth and health issues, divided into 10 typical disorder categories: Attention Deficit, Tics, Developmental Delays, Autism Spectrum, Anxiety, Anorexia, Obsessive-Compulsive Disorder, Sleep Issues, Early Puberty, and Depressive Disorders. These questions were then posed to various large language models. A pediatric specialist evaluated the models' answers using a detailed four-tier system (ranging from Poor to Very Good) for accuracy. To ensure consistency, these assessments were revisited and verified at various intervals. High-scoring responses ('Good' or above) were examined closely for their compliance with medical ethics, treatment guidelines, and diagnostic procedures. In contrast, responses that scored lowest ('Poor') were subject to in-depth review, leading to recommendations for minor modifications based on straightforward query adjustments and online medical resources. These revised responses were then re-evaluated to measure any improvements in accuracy. Findings: Our study analyzed the performance of different models in adolescent growth and development issues. Claude2 was the top performer, with an average score of 3.54 and a standard deviation of 0.501. ChatGPT-3.5 was close behind, scoring an average of 3.44 and a standard deviation of 0.519. Human raters and Google Bard scored lower, at 2.60 and 2.49 respectively, with larger standard deviations. The one-way ANOVA showed significant differences (F-value 64.692, P-value 4.64e-34), particularly in areas like 'Attention Deficit Disorder', 'Developmental Delay', and 'Depression', where Claude2 and ChatGPT-3.5 outperformed others. The Pearson Chi-Square test (χ² value 117.758, P-value 2.35e-25) confirmed their accuracy and consistency. In self-correction abilities, Claude2, ChatGPT-3.5, and Bard scored 3.3, 3.0, and 2.4, respectively, for simple query-based corrections. For web-based medical self-corrections, the scores improved to 3.8, 3.5, and 3.7. The Pearson Chi-Square tests showed significant improvements for all models (Claude2 P-value 0.0241, ChatGPT-3.5 P-value 0.0150, Bard P-value 0.000017), with Bard showing the most significant improvement. This indicates that web-based medical correction methods significantly enhance performance in complex queries for all LLM chatbots. Interpretation: Our findings underscore the potential of Large Language Models (LLMs), particularly Claude2, in providing accurate and comprehensive responses to queries related to adolescent growth and development. The continual strategies and evaluations to enhance the accuracy of LLMs remain crucially important.