Information amount, accuracy, and relevance of generative artificial intelligences’ answers to learning objectives of medical arthropodology evaluated in English and Korean queries in December 2023: a descriptive study

Abstract

Purpose: This study aimed to assess 6 generative artificial intelligences (AIs)' performance on the learning objectives of medical arthropodology in parasitology class in Korea. We examined AIs' performance by querying in Korean and English to determine their information amount, accuracy, and relevance in both language prompts. Methods: From December 15 to 17, 2023, 6 generative AIs, including Bard, Bing, Claude, Clova X, GPT-4, and Wrtn, were tested on 7 medical arthropodology learning objectives in English and Korean. Clova X and Wrtn are platforms from Korean companies. Responses were evaluated for the criteria in English and Korean queries. Results: Bard had abundant information but was fourth in accuracy and relevance. GPT-4, with high information content, ranked first in accuracy and relevance. Clova X was 4th in amount but 2nd in accuracy and relevance. Bing provided lower pieces of information with moderate accuracy and relevance. Wrtn's answer was short of data, with average accuracy and relevance. Claude AI had reasonable information but lower accuracy and relevance. Responses in English were superior in all aspects. Clova X was notably optimized for Korean, leading in relevance.Conclusion: In a study of 6 generative AIs applied to medical arthropodology, GPT-4 excelled overall, while Clova X, a Korea-based AI, achieved 100% relevance in Korean queries, the highest among its peers. Utilizing these AIs in classrooms enhanced the author's self-efficacy and interest in the subject, offering a positive experience of interacting with generative AIs to question and receive information.