IRT-Based Assessments of Rater Effects in Multiple-Source Feedback Instruments

Abstract

This study compares three item response theory–based models of assessing measurement equivalence in 360° feedback: the traditional differential item functioning (DIF) methodology, Muraki’s rater’s effect model, and Patz, Junker, and Johnson’s hierarchical rater model. Using data from 491 managers collected on the Benchmarks instrument, the authors found that the traditional DIF methodology provides the most information about the rater’s conception of the ratee’s ability, whereas the other two models provide explicit estimates of rater leniency/severity. The authors also found that rater source effects of leniency and severity, even though statistically significant, did not substantially affect the observed score at the item and scale levels. The different results and conclusions produced by each model are discussed.