Tıp Eğitiminde Otomatik Soru Üretme Yöntemi Kullanılarak Oluşturulan İlk Türkçe Çoktan Seçmeli Soruların Psikometrik Analizi

Abstract

Aim: Automatic item generation is "a process of using models to generate items using computer technology". The use of automatic item generation typically involves one of three primary methods: syntax-based, semantic-based, and template-based. Non-template automatic item generation approaches leverage natural language processing techniques. A study showed the potential of using template-based automatic item generation to create high-quality multiple-choice questions for assessing clinical reasoning in Turkish, marking a first in the field. However, the findings of the study were based only on expert opinions, necessitating further research to examine the psychometric qualities of Turkish items. The aim of this study was to reveal psychometric characteristics of the first Turkish case-based multiple-choice questions generated by using automatic item generation in medical education. Methods: This was a psychometric study. Three Turkish case-based multiple-choice questions generated using template-based automatic item generation on essential hypertension were included in an exam that 281 fourth-year medical students participate in. This examination was carried out in-person in classroom settings under proctor supervision. Item difficulty and item discrimination (point-biserial correlation) were calculated, and non-functioning distractors were determined. Results: All three items had acceptable levels (higher than 0.20) of point-biserial correlation (p<0.001). The item difficulty levels indicated the presence of one easy, one moderate, and one difficult question. Each item had 2-3 non-functioning options among five options. All three items had acceptable levels (higher than 0.20) of point-biserial correlation (p<0.001). The item difficulty levels indicated the presence of one easy, one moderate, and one difficult question. Each item had 2-3 non-functioning options among five options. Conclusions: The results indicated that the items successfully discriminate between high and low performers, providing validity evidence on the quality of the questions in evaluating students' comprehension of the subject. Additionally, the findings suggest that it is feasible to create multiple-choice questions with different difficulty levels in Turkish using a single automatic item generation model. This study demonstrated for the first time that automatic generation of case-based multiple-choice questions in Turkish produces acceptable psychometric characteristics in an authentic assessment setting in medical education. The ability to automatically generate effective multiple-choice questions in Turkish holds promise for enhancing the efficiency of written assessment in Turkish medical education.