Confounding among Measures of Leniency and Halo

Abstract

Because the variability of rating distributions decreases with leniency, common indices of the psychometric performance of rating data may be confounded. Lenient rating distributions, for example, should show lower average scale intercorrelations (a measure of halo) than do less lenient rating distributions. Although psychometric theory would predict such confounding, it does not permit the estimation of the magnitude of this phenomenon. Consequently, the interrelationships among halo and leniency measures were examined using simulated rating data. Specifically, effects of leniency on 3 levels of halo were examined. As expected, results showed that as leniency increased, halo decreased when measured by dimension intercorrelations but increased when measured by standard deviations across dimensions. The magnitude of the convergent validities calculated for halo and leniency measures also decreased as leniency increased. In addition, although halo and leniency were orthogonal when no leniency was present, substantial confounding of these constructs occurred as leniency increased. Implications of these results for the use and integration of the various measures are discussed.