The performance of large language models in managing abnormal results of cervical cancer screening: Comparative Study (Preprint)

Abstract

Cervical cancer remains the fourth leading cause of female death globally. Screening for cervical cancer is an effective preventative strategy. However, its impact is lessened in environments with scarce medical resources due to poor clinical decision-making and improper resource allocation. Large Language Models (LLMs) could significantly enhance medical systems in these settings by improving decision-making processes.

This study aims to evaluate the performance of LLMs in managing abnormal cervical cancer screening results.

Models are selected from AlpacaEval leaderboard version 2.0 and the capability of our computer. Questions inputted to models are designed in accordance to CSCCP and ASCCP guidelines. Two experts review the response from each model for accuracy, guideline compliance, clarity, and practicality by grading as A, B, C and D weighted as 3, 2, 1 and 0 scores, respectively. Effective rate is calculated as the ratio of the number of A and B to the number of all designed questions.

Nine models are included in this study, while 33 questions are specifically designed. Seven models (ChatGPT 4.0 Turbo, Claude 2, Gemini Pro, Mistral-7B-v0.2, Starling-LM-7B alpha, HuatuoGPT and BioMedLM 2.7B) provide stable responses. Among all included models, ChatGPT 4.0 Turbo and Claude 2 ranked in first level with mean score 2.30[2.0, 2.60] (effective rate: 88.81%) and 2.21[1.88, 2.54] (effective rate: 78.79%) when compared to the other seven models (P<0.001).

Proprietary LLMs, particularly ChatGPT 4.0 Turbo and Claude 2, show promise in clinical decision-making involving logical analysis. However, this study underscores the need for further research to explore the practical application of LLMs in medicine.