Development and Evaluation of Shortcut-Derived Behaviorally Anchored Rating Scales

Abstract

The purpose of the present study was to evaluate a shortcut method for developing Behaviorally Anchored Rating Scales (BARS). The developmental method included two novel steps: (a) the performance of judgmental cluster analysis to define scales and (b) the direct placement of behavioral incidents on the scales rather than the use of the retranslation step. Two groups of six students each developed shortcut BARS through using an instructional booklet. The BARS were developed to evaluate teaching performance of college instructors. The sets of shortcut scales constructed by the two groups and a traditionally developed BARS were then distributed randomly to students in nine classes who were asked to evaluate both their class instructors and the instructors in their immediately preceding class. The shortcut scales, in comparison with the traditional BARS, were slightly more reliable. Both sets of scales had approximately the same degree of leniency error. The shortcut scales were in general somewhat less adequate in terms of halo error and discriminant validity, but were almost equivalent on these characteristics after the scales with very low reliabilities had been dropped. The results did not support the use of the judgmental cluster analysis, but did indicate that the retranslation step may not be necessary to develop BARS.