Assessing Large Language Models’ Proficiency, Clarity, and Objectivity at the Intersection of Obstetrics, Gynecology, and Global Public Health: Cross-Sectional, Comparative Analysis with Specialists' Knowledge on COVID-19 Impacts in Pregnancy (Preprint)

Abstract

The COVID-19 pandemic has significantly strained healthcare systems globally, leading to an overwhelming influx of patients and exacerbating resource limitations. Concurrently, an “infodemic” of misinformation, particularly prevalent in women's health, has emerged. This challenge has been pivotal for healthcare providers, especially gynecologists and obstetricians, in managing pregnant women's health. The pandemic heightened risks for pregnant women from COVID-19, necessitating balanced advice from specialists on vaccine safety versus known risks. Additionally, the advent of generative Artificial Intelligence (AI), such as large language models (LLMs), offers promising support in healthcare. However, they necessitate rigorous testing.

To assess LLMs’ proficiency, clarity, and objectivity regarding COVID-19 impacts in pregnancy.

This study evaluates four major AI prototypes (ChatGPT-3.5, ChatGPT-4, Microsoft Copilot, and Google Bard) using zero-shot prompts in a questionnaire validated among 172 Israeli gynecologists and obstetricians. The questionnaire assesses proficiency in providing accurate information on COVID-19 in relation to pregnancy. Text-mining, sentiment analysis, and readability (Flesch-Kincaid grade level) were also conducted.

In terms of LLMs’ knowledge, ChatGPT-4 and Microsoft Copilot each scored 96.7%, Google Bard 93.3%, and ChatGPT-3.5 80.0%. Concerning misinformation instances, ChatGPT-4 incorrectly stated an increased risk of miscarriage due to COVID-19. Google Bard and Microsoft Copilot had minor inaccuracies concerning COVID-19 transmission and complications. At the sentiment analysis, polarity scores were moderately positive, with ChatGPT-4 at 0.37, followed by Microsoft Copilot at 0.33, ChatGPT-3.5 at 0.25, and Google Bard at 0.23. Subjectivity levels were moderate, with Microsoft Copilot being the most objective (0.42). Finally, concerning the readability analysis, Flesch-Kincaid Grade Level showed ChatGPT-3.5 at 25.34, followed by Google Bard at 18.30, Microsoft Copilot at 11.27, and ChatGPT-4 at 21.12.

The study highlights varying knowledge levels of LLMs in relation to COVID-19 and pregnancy. ChatGPT-3.5 showed the least knowledge and alignment with scientific evidence. Readability and complexity analyses suggest that each AI's approach is tailored to specific audiences, with ChatGPT versions being more suitable for specialized readers. The sentiment analysis underscores the importance of factual and objective information dissemination. Overall, ChatGPT-4, Microsoft Copilot, and Google Bard generally provide accurate, updated information on COVID-19 and vaccines in women's health, aligning with health guidelines. The study demonstrates the potential role of AI in supplementing healthcare knowledge, with a need for continuous updating and verification of AI knowledge bases. The choice of AI tool should consider the target audience and required information detail level.