The role of implementation in generalisability: A synthesis of evidence on targeted educational instruction and a new randomised trial

Abstract

Targeted instruction is one of the most effective educational interventions in low- and middle-income countries, yet the reported impacts of this approach vary, from 0.07 to 0.78 standard deviations (SDs) across contexts. We study this variation and the contextual factors associated with it by combining an evidence aggregation covering 10 study arms with a new randomised trial. The results show that two factors explain most of the heterogeneity in reported effects: the degree of implementation (intention-to-treat or treatment-on-the-treated effects) and the instruction delivery model (teachers or volunteers). Accounting for these implementation factors enables substantial generalisation of effect sizes across contexts. We introduce a new Bayesian model which incorporates implementation information into the evidence aggregation process. The results show that targeted instruction can deliver 0.39 SD improvements in learning on average when taken up, and 0.80 SD gains when implemented with high fidelity, explaining the upper range of effects in the literature. Given the central role of implementation identified in our synthesis, we conduct a new randomised trial to increase programme fidelity in Botswana. The results show additional 0.22 SD gains relative to standard implementation, revealing concrete mechanisms to enhance implementation and achieve the largest frontier effects identified in the literature.