Simultaneous Estimation of Overall and Domain Abilities: A Higher-Order IRT Model Approach

Abstract

Assessments consisting of different domains (e.g., content areas, objectives) are typically multidimensional in nature but are commonly assumed to be unidimensional for estimation purposes. The different domains of these assessments are further treated as multi-unidimensional tests for the purpose of obtaining diagnostic information. However, when the domains are disparate, assuming a single underlying ability across the domains is not tenable. Moreover, estimating domain proficiencies based on short tests can result in unreliable scores. This article presents a higher-order item response theory framework where an overall and multiple domain abilities are specified in the same model. Using a Markov chain Monte Carlo method in a hierarchical Bayesian framework, the overall and domain-specific abilities, and their correlations, are estimated simultaneously. The feasibility and effectiveness of the proposed model are investigated under varied conditions in a simulation study and illustrated using actual assessment data. Implications of the model for future test analysis and ability estimation are also discussed. Index terms: higher-order ability estimation, item response theory, multidimensionality, domain scoring, Markov chain Monte Carlo