Estimating the Distribution of Treatment Effects From Random Design Experiments

Abstract

Background: Random design experiments are a powerful device for estimating average treatment effects, but evaluators sometimes seek to estimate the distribution of treatment effects. For example, an evaluator might seek to learn the proportion of treated units who benefit from treatment, the proportion who receive no benefit, and the proportion who are harmed by treatment. Method: Imbens and Rubin (I&R) recommend a Bayesian approach to drawing inferences about the distribution of treatment effects. Drawing on the I&R recommendations, this article explains the approach; provides computing algorithms for continuous, binary, ordered and countable outcomes; and offers simulated and real-world illustrations. Results: This article shows how the I&R approach leads to bounded uncertainty intervals for summary measures of the distribution of treatment effects. It clarifies the nature of those bounds and shows that they are typically informative. Conclusions: Despite identification issues, bounded solutions provide useful insight into the distribution of treatment effects. We recommend that evaluators incorporate analyses of the distribution of treatment effects into new studies and that evaluators revisit completed studies to estimate the distribution of treatment effects.