Methodologically Based Discrepancies in Compensatory Education Evaluations

Abstract

Meta-analysis of several hundred evaluations of Title I compensatory education programs shows that two distinct research designs consistently yield different results. The norm- referenced model portrays programs as positively effective while the regression-disconti nuity design shows them to be ineffective or even slightly harmful. Three potential biasing factors are discussed for each design—residual regression artifacts; attrition and time-of- testing problems in the norm-referenced design; and assignment, measurement, and data preparation problems in the regression-discontinuity design. In lieu of more definitive research the tentative conclusion is that in practice the norm-referenced design over estimates the program effect while the regression-discontinuity design underestimates it.