Validation of the QAMAI tool to assess the quality of health information provided by AI

Abstract

AbstractObjectiveTo propose and validate the Quality Assessment of Medical Artificial Intelligence (QAMAI), a tool specifically designed to assess the quality of health information provided by AI platforms.Study designobservational and valuative studySetting27 surgeons from 25 academic centers worldwide.MethodsThe QAMAI tool has been developed by a panel of experts following guidelines for the development of new questionnaires. A total of 30 responses from ChatGPT4, addressing patient queries, theoretical questions, and clinical head and neck surgery scenarios were assessed. Construct validity, internal consistency, inter-rater and test-retest reliability were assessed to validate the tool.ResultsThe validation was conducted on the basis of 792 assessments for the 30 responses given by ChatGPT4. The results of the exploratory factor analysis revealed a unidimensional structure of the QAMAI with a single factor comprising all the items that explained 51.1% of the variance with factor loadings ranging from 0.449 to 0.856. Overall internal consistency was high (Cronbach’s alpha=0.837). The Interclass Correlation Coefficient was 0.983 (95%CI 0.973-0.991; F(29,542)=68.3;p<0.001), indicating excellent reliability. Test-retest reliability analysis revealed a moderate-to-strong correlation with a Pearson’s coefficient of 0.876 (95%CI 0.859-0.891;p<0.001)ConclusionsThe QAMAI tool demonstrated significant reliability and validity in assessing the quality of health information provided by AI platforms. Such a tool might become particularly important/useful for physicians as patients increasingly seek medical information on AI platforms.