The Effect of Number of Rating Scale Categories on Levels of Interrater Reliability : A Monte Carlo Investigation

Abstract

A computer simulation study was designed to in vestigate the extent to which the interrater reliability of a clinical scale is affected by the number of cate gories or scale points (2, 3, 4, ... ,100). Results in dicate that reliability increases steadily up to 7 scale points, beyond which no substantial increases occur, even when the number of scale poirits is increased to as many as 100. These findings hold under the follow ing conditions: (1) The research investigator has insuf ficient a priori knowledge to use as a reliable guide line for deciding on an appropriate number of scale points to employ, and (2) the dichotomous and ordinal categories being considered all have an underlying metric or continuous scale format.