A Framework for Developing Cognitively Diagnostic Assessments

Abstract

Over the past decade or so, a growing number of writers have argued that cognitive science and psychometrics could be combined in the service of instruction. Researchers have progressed beyond statements of intent to the hands-on business of researching and developing diagnostic assessments combining cognitive science and psychometrics, what I call cognitively diagnostic assessment (CDA). In this article, I attempt to organize the many loosely connected efforts to develop cognitively diagnostic assessments. I consider the development of assessments to guide specific instructional decisions, sometimes referred to as diagnostic assessments. Many of my arguments apply to program evaluation as well—assessments that reveal the mechanisms test takers use in responding to items or tasks provide important information on whether instruction is achieving its goals. My goal in this article is to characterize CDA in terms of the intended use of assessment and the methods of developing and evaluating assessments. Towards this goal, I (a) outline the societal trends that motivate the development of CDA, (b) introduce a framework within which the psychological and statistical aspects of CDA can be coordinated, and (c) summarize efforts to develop CDA in a five-step methodology that can guide future development efforts. Finally, I address some of the issues developers of CDA must resolve if CDA is to succeed.