The Effects of Different Rater Training Procedures on ESL Essay Raters’ Rating Accuracy

Abstract

The study investigated the effects of three commonly employed rater training procedures on the rating accuracy of novice ESL essay raters. The first training procedure involved going through a set of benchmarked scripts with scores, the second involved assessing benchmarked scripts before viewing the scores. The third was a combination of the former and latter. A pre, post and delayed post-experimental research design was employed. Data were collected before, immediately after and one month after training. Actual IELTS scripts with benchmarked scores determined by subjecting expert IELTS raters’ scores through Multi-Faceted Rasch (MFR) analysis were used for the training and data collection purposes. Sixty-three TESL trainees were selected based on their pre-training rating accuracy to form three equally matched experimental groups. The trainees’ scores for the essays before, immediately after and one month after the assigned training procedure were compared with the official scores for the operational essays. Although the findings indicate that generally, rater training improves raters’ rating accuracy by narrowing the gap between their scores and the official scores, different training procedures seem to have different effects. The first training procedure significantly improved raters’ rating accuracy but showed a decreasing effect with time. The second training procedure showed immediate as well as delayed positive effects on raters’ rating accuracy. The third training did not lead to significant immediate improvement, but rating accuracy improved significantly after some time. This paper discusses the implications of the findings in planning efficient and effective rater training programmes.