The Effect of Aberrant Responses on Ability Estimation in Computer Adaptive Tests

Abstract

In computer adaptive testing (CAT), aberrant responses caused by some factors such as lucky guesses and carelessness errors may cause significant bias on ability estimation. Correct responses resulting from lucky guesses and false responses resulting from carelessness or anxiety may reveal aberrant responses and the impact of these types of aberrant responses may cause an erroneous estimation of the examinee’s actual ability because they do not reflect the examinee’s actual knowledge. In this study, the performances of 3PL and 4PL models regarding ability estimation were examined comparatively in the context of CAT simulations in case of aberrant responses. Under different conditions, twelve separate CAT simulations were conducted with 10 replications for each of the condition. Correlation, RMSE, bias, and mean absolute error (MAE) values were calculated and interpreted for each condition. Results generally indicated that the 4PL model provided more efficient and robust ability estimation than the 3 PL model and the 4PL model increased the precision and effectiveness of the CAT applications.