Use of Rasch Modeling and Focus Groups to Inform the Training of Teacher Evaluators

Abstract

There are few empirical studies of teacher performance evaluation systems. Teachers are rightfully concerned about the degree to which evaluators’ idiosyncratic biases might undermine the process. Training evaluators thoroughly and monitoring the reliability, validity, fairness, and cultural sensitivity of their ratings are essential steps towards promoting strong performance evaluation systems. This study examined the process of evaluating early childhood teachers to inform evaluator training. The researchers sought to determine the degree to which the expectations of those who develop training materials and conduct evaluator trainings differ from the typical performance ratings given by evaluators in the field. Researchers used several methods to prompt a systematic examination of the evaluator training process across four sequential phases of investigation: (a) quantitative panel ratings of item difficulty, (b) panel discussion and consensus building (a qualitative phase), (c) examining expected versus empirical item difficulty (a quantitative phase), and (d) presenting the empirical difficulty levels to the panel for discussion (a qualitative phase). In this last phase, researchers presented results of Rasch modeling to the panel, along with levels of agreement between the empirical and expected difficulty levels. Panel members reported that the process of discussing their perceptions of expected item difficulty levels was valuable. They also reported that such discussion prompted them to reevaluate the training materials, the resource manuals, and other professional development resources. The study methods presented can be used to investigate and to improve other personnel evaluation systems.