Inside the Black Box: Examining Mediators and Moderators of a Middle School Science Intervention

Abstract

We use data from a randomized controlled trial of a middle school science intervention to explore the causal mechanisms by which the intervention produced previously documented gains in student achievement. Our study finds that implementation fidelity, operationalized as a measure of the frequency of implementation of the cognitive science principles taught in intervention teachers’ professional development, helped to explain student achievement effects. Integrating findings from a structural equation model using data from 10,281 students and 124 teachers with a small subsample of teacher interviews, we also found that the intervention may work partly through fostering better classroom management and collaborative discussions that elevated practice. Furthermore, our results have implications for informing decisions about how to balance a focus on increasing teacher content knowledge, on one hand, and providing explicit pedagogical strategies linked to the curriculum, on the other. We additionally found that lower achieving classrooms had lower implementation scores, likely due to factors that hindered teachers’ ability to use the cognitive science principles, such as less science class time or needing to adapt content to meet the needs of struggling students. Our study highlights the importance of anticipating—and calibrating interventions to—the contextual complexities of real-life classrooms, and it identifies several factors with the potential to contribute to improved design and evaluation of such interventions.