Toward the Relational Management of Educational Measurement Data

Abstract

Background Historically, significant advances in scientific understanding have followed advances in measurement and observation. As the resolving power of an instrument increased, so have gains in the understanding of the phenomena being observed. Modern interactive systems are potentially the new “microscopes” when they are instrumented to record fine-grained observations of what students do in an online task. Advances in the conceptualization, design, and analyses of such interaction data enable the discovery of learning patterns and can power new applications. One application, personalization, is one of 14 engineering Grand Challenges identified by the National Academy of Engineering (2008). Purpose This article examines three levels of data available in online systems that can be used to understand student performance. Empirical research is reviewed to examine three fundamental questions: To what extent does students’ online behavior (a) relate to their cognitive processing, and to what extent can student behavior (b) be used to model their problem solving process and (c) be used diagnostically to reveal understandings and misconceptions? Participants The reviewed studies involved participants from college and K12 settings. Research Design The reviewed studies all focused on learning processes and outcomes. Nearly all studies had high frequency process-tracing data such as concurrent think-alouds, moment-to-moment telemetry, or both. Participants interacted with an online task on an academic subject. The task typically spanned one or a few class periods, and the studies collectively examined relations among students’ online behavior, cognitive processes (via think-alouds), and external measures of learning. Data Collection and Analysis In the reviewed studies, students’ online behavior was captured by instrumenting the system to capture and log interaction events. The more sophisticated approaches used a telemetry design based on the presumed cognitive processing occurring in the system. Findings In general, measures derived from students’ online behavior can be used (a) to decide when to intervene to influence learning processes (e.g., increased help seeking) and outcomes (e.g., improved course grades) and (b) as proxy measures of cognitive processing, understanding, and misconceptions. Conclusions In the coming years, multiple levels of data will be fused to better understand the student, including static data such as demographics, low-frequency data such as interactions within a learning management system, and high frequency data such as moment-to-moment interactions in a digital app. As education enters the era of big data and transmedia-based learning, data of and for an individual will power new applications to realize the promise of personalized instruction.