The Identification of Guessing Patterns in Progress Testing as a Machine Learning Classification Problem

Abstract

Background The detection of guessing patterns in low-stakes progress testing could naturally be understood as a statistical classification problem where test takers are assigned to groups according to probabilities given by a machine learning model. However, the relevant literature on this topic does not include many examples where this approach is discussed; to date, the strategies applied to tackle this problem have been mostly based either on rapid response counting or the detection of unusual answer patterns. Methods On the basis of 14,897 participations in the Progress Test Medizin test – which takes place twice a year since 1999 in selected medical schools of Germany, Austria and Switzerland - we formulate the identification of guessing patterns as a binary classification problem. Next, we compare the performance of a logistic regression algorithm in this setup to that of the nonparametric person-fit indices included in R´s PerFit package. Finally, we determine probability thresholds based on the values of the logistic regression functions obtained from the algorithm. Results Comparison of logistic regression algorithm with person-fit indices The logistic regression algorithm included in Python´s Scikit-Learn reached ROC-AUC scores of 0.886 to 0.903 depending on the dataset, while the 11 person-fit indices analysed returned ROC-AUC scores of 0.548 to 0.761. Best feature set Datasets based on aggregate scores yielded better results than those were the sets of answers to every item were considered as individual features. The best results were reached with a feature set containing only two parameters (self-monitoring accuracy and number of answered questions); considering the amount of time spent on the test did not lead to any performance improvement. Probability thresholds Based on the values of the logistic regression function generated by the applied algorithm, it is possible to establish thresholds above which there is at least a 90% chance of having guessed most answers. Conclusions In our setting, logistic regression clearly outperformed nonparametric person-fit indices in the task of identifying guessing patterns. We attribute this result to the greater flexibility of machine learning methods, which makes them more adaptable to diverse test environments than person-fit indices.