Evaluating the effectiveness of a mathematics bridge program using propensity scores

Abstract

Purpose – The purpose of this paper is to illustrate how to examine the effectiveness of a pilot summer bridge program for elementary algebra using propensity scores. Typically, selection into treatment programs, such as summer bridge programs, is based on self-selection. Self-selection makes it very difficult to estimate the true treatment effect because the selection process itself often introduces a source of bias. Design/methodology/approach – By using propensity scores, the authors can match students who participated in the summer bridge program with equivalent students who did not participate in the summer bridge program. By matching students in the treatment group to equivalent students who do not participate in the treatment, the authors can obtain an unbiased estimate of the treatment effect. The authors also describe a method to conduct a sensitivity analysis to estimate the amount of hidden bias generated from unobserved factors that would be needed to alter the inferences made from a propensity score matching analysis. Findings – Findings suggest there is no significant difference in the pass rates of the subsequent intermediate algebra course for students who participated in the summer bridge program when compared to matched students who did not participate in the summer bridge program. Thus, students who participate in the summer bridge program fared no better or worse when compared to similar students who do not participate in the program. These findings also appear to be robust to hidden bias. Originality/value – This study describes a unique way to estimate the causal effect of participating in a treatment program when there is self-selection into the treatment program.