Comparison of the Knowledge of Large Language Models and General Radiologist on RECIST (Preprint)

Abstract

Large language models (LLMs) represent a remarkable breakthrough in natural language processing. What sets the current generation of LLMs apart is their remarkable ability to perform very specific tasks in radiology, as in many other fields, without the need for additional training. LLMs have the potential to usher in a new era of efficiency and excellence in radiology practice, both in their potential as supportive diagnostic tool and in their ability to facilitate the reporting process. This great potential of LLMs is of great importance in both oncology and oncological radiology, and researchers have been conducting many new studies in these fields in order to demonstrate the position of LLMs.

We aimed to provide a new perspective on their potential to facilitate reporting and improve reporting efficiency in oncological imaging by comparatively assessing LLMs' knowledge of RECIST 1.1 both among themselves and with general radiologist (GR).

Radiologist (E.Ç.) prepared the 25 multiple-choice questions in this study utilizing the information in RECIST 1.1, thus eliminating the need for ethics committee approval. We initiated the input prompt as follows: ‘‘Act like a professor of radiology who has 30 years of experience in oncological radiology.. Give just letter of the most correct choice of multiple choice questions. Each question have only one correct answer.’’This prompt was tested in June 2024 on seven different LLMs using the default settings. The testing included models from various developers: Claude 3 Opus, ChatGPT 4 and ChatGPT 4o, Gemini 1.5 pro, Mistral Large, Meta Llama 3 70B, Perplexity pro. Also GR (T.C.) board certified by EDiR and with 6 years of experience each, answered the same questions

The results revealed that Claude 3 Opus achieved the highest accuracy of 100% (25/25 questions), followed by newest model of Open AI’s ChatGPT 4o with 96% accuracy (24/25 questions). ChatGPT 4 and Mistral Large 92% (23/25 questions), Meta Llama 3 70 B, Perplexity pro and Gemini 1.5 pro had accuracy of 88% (21/25 questions). GR (T.C.) has accuracy of 92% (23/25 questions).

The outstanding success of Claude 3 Opus by knowing all the questions raises the question of whether oncology can be a new star among LLMs in radiology. Our study reveals that the majority of LLM models exhibit a commendable level of proficiency and comperable to GR in answering RECIST 1.1 related questions. Our findings show that current LLM models have more than sufficient text-based information about RECIST 1.1. Additionally, our findings underscore the high potential of LLMs as tools to assist radiologists in oncology reporting. However, to take full advantage of LLMs' abilities in oncology reporting, it is of great importance that their visual abilities are also evaluated. Visual evaluation forms the basis of radiology. Therefore, future studies should focus on evaluating the visual information of multimodal LLMs with visual evaluation ability. In conclusion, LLMs have great potential in oncological radiology as in every field of radiology, and new studies that will reveal this potential will allow LLMs to be more easily integrated into radiological practice.

There is no need to trial registration.

RR2-10.2196/preprints.64805