A replication recipe: list your ingredients before you start cooking

Abstract

Abstract The author argues that researchers should do replications using preanalysis plans. These plans should specify at least three characteristics that would act as stopping rules for the replicator: (1) how much flowtime the replicator will spend, (2) how much money and effort (working hours) the replicator will spend, and (3) the intended results and the precision of the replication necessary for “success”. A researcher’s replication will be “successful” according to context-specific criteria in the preanalysis plan. The author also argues that the two biggest drawbacks of preanalysis plans—(1) that they discount unexpected but extraordinary findings and (2) that they make it difficult for researchers to prespecify all possible actions in their decision trees—are less relevant for replications compared with new research. He concludes with describing a preanalysis plan for replicating a paper on housing demand and household formation.