Large language models to differentiate vasospastic angina using patient information

Abstract

AbstractBackgroundVasospastic angina is sometimes suspected from patients’ medical history. It is essential to appropriately distinguish vasospastic angina from acute coronary syndrome because its standard treatment is pharmacotherapy, not catheter intervention. Large language models have recently been developed and are currently widely accessible. In this study, we aimed to use large language models to distinguish between vasospastic angina and acute coronary syndrome from patient information and compare the accuracies of these models.MethodWe searched for cases of vasospastic angina and acute coronary syndrome which were written in Japanese and published in online-accessible abstracts and journals, and randomly selected 66 cases as a test dataset. In addition, we selected another ten cases as data for few-shot learning. We used generative pre-trained transformer-3.5 and 4, and Bard, with zero- and few-shot learning. We evaluated the accuracies of the models using the test dataset.ResultsGenerative pre-trained transformer-3.5 with zero-shot learning achieved an accuracy of 52%, sensitivity of 68%, and specificity of 29%; with few-shot learning, it achieved an accuracy of 52%, sensitivity of 26%, and specificity of 86%. Generative pre-trained transformer-4 with zero-shot learning achieved an accuracy of 58%, sensitivity of 29%, and specificity of 96%; with few-shot learning, it achieved an accuracy of 61%, sensitivity of 63%, and specificity of 57%. Bard with zero-shot learning achieved an accuracy of 47%, sensitivity of 16%, and specificity of 89%; with few-shot learning, this model could not be assessed because it failed to produce output.ConclusionGenerative pre-trained transformer-4 with few-shot learning was the best of all the models. The accuracies of models with zero- and few-shot learning were almost the same. In the future, models could be made more accurate by combining text data with other modalities.