Assessing Dimensionality of Noncompensatory Multidimensional Item Response Theory With Complex Structures

Abstract

The purpose of this study was to investigate the effect of complex structure on dimensionality assessment in noncompensatory multidimensional item response models using dimensionality assessment procedures based on DETECT (dimensionality evaluation to enumerate contributing traits) and NOHARM (normal ogive harmonic analysis robust method). Five methods were evaluated: two DETECT-based methods—exploratory and cross-validated—and three NOHARM-based methods: root mean square residual (RMSR), [Formula: see text], and approximate likelihood ratio (ALR). The results suggested that the studied methods had varying degree of success in correctly counting the number of dimensions and in meaningfully labeling sets of items as dimension-like. In two-dimensional, shorter tests, [Formula: see text] and ALR largely outperformed RMSR- and DETECT-based methods in conditions with small and medium sample sizes, across all levels of complexity and correlations. Lengthening of the test in two-dimensional conditions led to most notably improved accuracy in determining the correct number of dimensions for NOHARM-based RMSR, whereas for the remaining methods, increase in the number of items had differential effect. DETECT-based methods, on the other hand, had more success in labeling sets of items as dimension-like in conditions with two dimensions, irrespective of the test length, suggesting the items that ought to be together were more often grouped together. Performance of the methods was further evaluated in conditions with larger dimensionality (i.e., 3) and conditions with the increased number of items (i.e., longer tests).