Many Replications Do Not Causal Inferences Make

Abstract

Although direct replications are ideal for randomized studies, areas of psychological science that lack randomized studies should incorporate Rosenbaum’s (2001) distinction between trivial and nontrivial replications, relabeled herein as exact and critical replications. If exact replications merely repeat systematic biases, they cannot enhance cumulative progress in psychological science. In contrast, critical replications distinguish between competing explanations by using crucial tests to clarify the underlying causal influences. We illustrate this potential with examples from research on corrective actions by professionals (e.g., psychotherapy, Ritalin) and parents (e.g., spanking, homework assistance), where critical replications are needed to overcome the inherent selection bias due to corrective actions being triggered by children’s symptoms. Purported causal effects must first prove to be replicable after plausible confounds such as selection bias are eliminated. Subsequent critical replications can then compare plausible alternative explanations of the average unbiased causal effect and of individual differences in those effects. We conclude that this type of systematic sequencing of critical replications has more potential for making the kinds of discriminations typical of cumulative progress in science than do exact replications alone, especially in areas where randomized studies are unavailable.