Experts’ prediction of the actual item difficulty of multiple-choice questions in the Ethiopian Undergraduate Medicine Licensure Examination

Abstract

Abstract Background The ability of expert ratings to predict the difficulty level of items to identify test-taker performance is an important aspect of licensure exams. Expert judgment is used as a primary source of information for users to make prior decisions to determine the pass rate of test takers. The nature of the raters involved in predicting item difficulty is central to setting credible standards. Therefore, this study aimed to assess and compare raters' prediction and actual MCQ item difficulty of the 2022 Ethiopian undergraduate medicine licensure examination (UGMLE). Method 200 Multiple-Choice Questions (MCQs) of the 2022 cohort of 815 UGMLE examinees were used in this study and seven physicians participated in the difficulty level ratings. Then, analysis was conducted to understand experts' rating variation in predicting the computed actual difficulty levels of examinees. Descriptive statistics to profile the rater’s assessment results and ANOVA to examine differences between the raters' estimations of the difficulty levels of the MCQs were computed. Additionally, regression analysis was used to understand the interrater variations in their predictions of difficult levels compared to actual difficult levels of MCQs.It was also used to examine the proportion of variation that each rater explained in the predictions of actual difficulty levels across the whole set of MCQs and all UGMLE fourteen domains. Results The study revealed statistically significant differences in the mean difficult level ratings of some raters and moderate to high positive linear relationships with some exam domains. But also, statistically nonsignificant relationships were found in some other domains. Thus, considerations have been needed on the observed variability in difficult-level rating values across raters and UGMLE domains. In the regression results, experts' ratings accounted for 33% of the variations in the actual UGMLE difficulty levels. Between the expert-rated and actual difficulty levels, the regression model showed a moderately positive linear correlation (R = 0.57) that was statistically significant at p = .05. Conclusion This study demonstrated the complex nature of rating the level of difficulty of MCQs on UGMLE and the benefits of employing prior expert ratings. To improve a rater's rating accuracy in UGMLE, designing strategies in line with the changing nature of assessment methods guarantees to maintain the required reliability and validity of the exam.