Comparing ChatGPT Responses with AHA Guidelines for Assessing Unruptured Intracranial Aneurysms: Establishment of a Simple Rating System

Abstract

Abstract Introduction Efficient diagnosis and intervention for unruptured intracranial aneurysms (UIAs) are crucial for favorable outcomes. Our study aimed to evaluate the accuracy and alignment of Chat Generative Pre-trained Transformer (ChatGPT) with established medical standards by systematically evaluating its responses using the American Heart Association (AHA) guidelines for the management of UIAs as a reference. This initiative bridges advanced artificial intelligence (AI) technology and medical practice norms, and contributes to the discussion on the role of AI in the dissemination of medical information. Methods In our collaborative study, we systematically assessed ChatGPT 3.5's responses by posing clinical questions aligned with AHA guidelines and evaluating them on a 1 to 5 scale for agreement and comprehensiveness. This method allowed us to objectively gauge ChatGPT's alignment with AHA medical guidelines. Results We introduced a set of ten clinical questions related to UIAs. Within this set, ChatGPT's responses achieved a 5-point rating for four questions. A further four questions were rated 3 points, and the remaining two questions received a score of 2. Conclusions By establishing a scoring system, we assessed the accuracy of ChatGPT responses to questions related to UIAs. It provides excellent results for screening, risk factors, and as a diagnostic tool. However, there is room for improvement in terms of the rupture risk and management.