Screening/diagnosis of pediatric endocrine disorders through the artificial intelligence model in different language settings

Abstract

AbstractThis study is aimed at examining the impact of ChatGPT on pediatric endocrine and metabolic conditions, particularly in the areas of screening and diagnosis, in both Chinese and English modes. A 40-question questionnaire covering the four most common pediatric endocrine and metabolic conditions was posed to ChatGPT in both Chinese and English three times each. Six pediatric endocrinologists evaluated the responses. ChatGPT performed better when responding to questions in English, with an unreliable rate of 7.5% compared to 27.5% for Chinese questions, indicating a more consistent response pattern in English. Among the reliable questions, the answers were more comprehensive and satisfactory in the English mode. We also found disparities in ChatGPT’s performance when interacting with different target groups and diseases, with improved performance for questions posed by clinicians in English and better performance for questions related to diabetes and overweight/obesity in Chinese for both clinicians and patients. Language comprehension, providing incomprehensive answers, and errors in key data were the main contributors to the low scores, according to reviewer feedback.Conclusion: Despite these limitations, as ChatGPT continues to evolve and expand its network, it has significant potential as a practical and effective tool for clinical diagnosis and treatment. What is Known:• The deep learning-based large-language model ChatGPT holds great promise for improving clinical practice for both physicians and patients and has the potential to increase the speed and accuracy of disease screening and diagnosis, as well as enhance the overall efficiency of the medical process. However, the reliability and appropriateness of AI model responses in specific field remains unclear.• This study focused on the reliability and appropriateness of AI model responses to straightforward and fundamental questions related to the four most prevalent pediatric endocrine and metabolic disorders, for both healthcare providers and patients, in different language scenarios. What is New:• The AI model performed better when responding to questions in English, with more consistent, as well as more comprehensive and satisfactory responses. In addition, we also found disparities in ChatGPT’s performance when interacting with different target groups and different diseases.• Despite these limitations, as ChatGPT continues to evolve and expand its network, it has significant potential as a practical and effective tool for clinical diagnosis and treatment.