Observational Studies versus Randomized Controlled Trials of Behavioral Interventions in Field Settings

Abstract

This article considers research designs that evaluate outcomes of behavioral interventions in field settings. It focuses on differences in efficacy estimates between observational studies (OSs) and randomized controlled trials (RCTs). The article contends that pretreatment motivation and in-treatment compliance both contribute to treatment outcomes. It proposes a 3-variable causal model in which pretreatment motivation produces positive treatment outcomes directly, and also indirectly, via in-treatment compliance. The article challenges the common notion that RCTs represent the gold standard in designs for evaluating the efficacy of behavioral interventions in field settings. The article's causal model predicts that OSs governed by self-assignment and RCTs of the same behavioral interventions both yield biased estimates of efficacy, although these effect-size biases are generally in opposite directions. OS estimates of efficacy are typically too large because of group differences in pretreatment motivation favoring the treated group over the untreated group. RCT efficacy estimates are typically too small because noncompliance in treatment conditions dilutes the impact of field interventions. Taken together, motivation and compliance thus account for the 2 expected efficacy biases: overestimation of effect sizes in OSs and underestimation in RCTs. Accordingly, the causal model predicts that, under most conditions, OSs will generate larger effect sizes than RCTs and thus a higher proportion of significant results. The article examines published outcome evaluations in 3 psychological domains: vocational counseling, precollege academic programs and home-visiting programs. Consistent with the model's directional prediction, studies in all 3 domains document a systematic efficacy difference between OSs and RCTs.