Developing and Using Indicators for Continuous Improvement

Abstract

Background/Context Amidst the complex and fast-paced demands in schools and classrooms, identifying what most deserves educators’ attention can pose a challenge. Indicators help focus attention by highlighting key features that signal important outcomes or opportunities to take action or learn. Purpose Informed by multiple literatures, we offer a conceptual framework that elaborates on the connection between an indicator's development and use. Specifically, the framework proposes that how to develop indicators depends on the answers to several questions about how they will be used: the intended use (why), timing (when), content (on what), and threshold (how much). Research Design To explore this conceptual framework, we present two cases on the construction and use of indicators for continuous improvement in the San Francisco Unified School District. In the first case, we applied statistical process control (SPC) theory to create control charts that visualize district-level outliers on student achievement outcomes to examine potential causes of variation among schools. In the second case, we combined SPC with more basic statistical methods to develop indicators highlighting areas of potential promise across multiple measures and student subgroups within a school. In our exploration of the two case studies, we present the statistical methods applied to construct the indicator, an example of the visualizations created, and descriptions of how central office leaders interpreted and used the results. We document how the intended use informed the indicators’ design, and the ramifications of those decisions for educators’ subsequent use of the indicators for improvement. Conclusions and Recommendations In learning from these cases, we highlight the importance of constructing indicators from the perspective of their intended use and linked with specific conditions and potential actions. This requires including more useful process data in the development of those indicators, as well as more follow-up analysis of the effectiveness of the actions taken based on those indicators, contrasted against alternative actions. We also underscore the importance of developing indicators that distinguish between the need for systemic versus targeted improvement efforts, based on SPC. We believe that combining approaches across disciplines and methodologies holds great promise for the development and use of indicators for continuous improvement in education.