Evaluating Methods for Estimating Program Effects

Abstract

I define a treatment effect in terms of a comparison of outcomes and provide a typology of all possible comparisons that can be used to estimate treatment effects, including comparisons that are relatively unknown in both the literature and practice. I then assess the relative merit, worth, and value of all possible comparisons based on the criteria of bias, precision, generalizability, ease of implementation, and cost. Which comparison is best depends on how these criteria are weighted and on the constraints of the specific research setting. I hope readers come to recognize a wider range of comparisons than they had previously, appreciate the value of considering all possible comparisons, and see how my typology of comparisons provides the basis for making fair appraisals of the relative strengths and weaknesses of different types of comparisons in the presence of the contingencies that are most likely to arise in practice.