Which tests not witch hunts: a diagnostic approach for conducting replication research

Abstract

Abstract Replication research can be used to explore original study results that researchers consider questionable, but it should also be a tool for reinforcing the credibility of results that are important to policies and programs. The challenge is to design a replication plan open to both supporting the original findings and uncovering potential problems. The purpose of this paper is to provide replication researchers with an objective list of checks or tests to consider when planning a replication study. The authors present tips for diagnostic replication exercises in four groups: validity of assumptions, data transformations, estimation methods, and heterogeneous impacts. For each group, the authors present an introduction to the issues, a list of replication tests and checks, some examples of how these checks are employed in replication studies of development impact evaluations, and a set of resources that provide statistical and econometric details. The authors also provide a list of don’ts for how to conduct and report replication research.