Comparison of Reliability Measures Under Factor Analysis and Item Response Theory

Abstract

Reliability of test scores is one of the most pervasive psychometric concepts in measurement. Reliability coefficients based on a unifactor model for continuous indicators include maximal reliability ρ and an unweighted sum score–based ω, among many others. With increasing popularity of item response theory, a parallel reliability measure π has been introduced using the information function. This article studies the relationship among the three reliability coefficients. Exploiting the equivalency between item factor analysis and the normal ogive model, π(2) for dichotomous data is shown to be always smaller than ρ. Additional results imply that ω is typically greater than π(2) in practical conditions, though mathematically there is no dominant relationship between π(2) and ω. Furtherresults indicate that, as the number of response categories increases, π can surpass ω. The reasons why π and ω fall short of ρ are also explored from an information gain/loss perspective. Implications of the findings on scale development and analysis are discussed.