Q-Matrix Refinement Based on Item Fit Statistic RMSEA

Abstract

A Q-matrix, which reflects how attributes are measured for each item, is necessary when applying a cognitive diagnosis model to an assessment. In most cases, the Q-matrix is constructed by experts in the field and may be subjective and incorrect. One efficient method to refine the Q-matrix is to employ a suitable statistic that is calculated using response data. However, this approach is limited by its need to estimate all items in the Q-matrix even if only some are incorrect. To address this challenge, this study proposes an item fit statistic root mean square error approximation (RMSEA) for validating a Q-matrix with the deterministic inputs, noisy, “and” (DINA) model. Using a search algorithm, two simulation studies were performed to evaluate the effectiveness and efficiency of the proposed method at recovering Q-matrices. Results showed that using RMSEA can help define attributes in a Q-matrix. A comparison with the existing Delta method and residual sum of squares (RSS) method revealed that the proposed method had higher mean recovery rates and can be used to identify and correct Q-matrix misspecifications. When no error exists in the Q-matrix, the proposed method does not modify the correct Q-matrix.