How Many Raters Should be Used for Establishing Cutoff Scores with the Angoff Method? a Generalizability Theory Study

Abstract

Although the Angoff method is perhaps the most highly regarded and widely used judgmental method for setting criterion-referenced cutoff scores, little research has been conducted to determine the optimal number of raters to employ for obtaining stable cutoff scores. In the present study, Angoff ratings obtained from eight different occupational licensing examinations were evaluated via generalizability theory to estimate the optimal number of raters. Results indicated that approximately 10 to 15 raters is an optimal target range, although fewer raters may sometimes be sufficient. It is recommended that 10 to 15 raters be sampled as a general rule, followed by analyses to determine the optimal number of raters for examinations in a particular profession using a particular procedural variant of the Angoff method.