The Reliability of the Posterior Probability of Skill Attainment in Diagnostic Classification Models

Abstract

One common score reported from diagnostic classification assessments is the vector of posterior means of the skill mastery indicators. As with any assessment, it is important to derive and report estimates of the reliability of the reported scores. After reviewing a reliability measure suggested by Templin and Bradshaw, this article suggests three new measures of reliability of the posterior means of skill mastery indicators and methods for estimating the measures when the number of items on the assessment and the number of skills being assessed render exact calculation computationally burdensome. The utility of the new measures is demonstrated using simulated and real data examples. Two of the suggested measures are recommended for future use.