Using a satisficing model of experimenter decision-making to guide finite-sample inference for compromised experiments

Abstract

Summary This paper presents a simple decision-theoretic economic approach for analysing social experiments with compromised random assignment protocols that are only partially documented. We model administratively constrained experimenters who satisfice in seeking covariate balance. We develop design-based small-sample hypothesis tests that use worst-case (least favourable) randomization null distributions. Our approach accommodates a variety of compromised experiments, including imperfectly documented rerandomization designs. To make our analysis concrete, we focus much of our discussion on the influential Perry Preschool Project. We reexamine previous estimates of programme effectiveness using our methods. The choice of how to model reassignment vitally affects inference.