Correction of Rater Effects in Longitudinal Research With a Cross-Classified Random Effects Model

Abstract

This study examines adverse consequences of using hierarchical linear modeling (HLM) that ignores rater effects to analyze ratings collected by multiple raters in longitudinal research. The most severe consequence of using HLM ignoring rater effects is the biased estimation of Levels 1 and 2 fixed effects and potentially incorrect significance tests about them. A cross-classified random effects model (CCREM) is proposed as an alternative to HLM. A Monte Carlo study and an empirical evaluation confirm that CCREM performs better than does HLM in dealing with rater effects. Strengths, limitations, and implications of the study are discussed.