The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse

Abstract

Even when guided by strong theories and sound methods, researchers must often choose a singular course of action from multiple viable alternatives. Regardless of the choice, it, along with all other choices made during the research process, individually and collectively affects study results, often in unpredictable ways. The inability to disentangle how much of an observed effect is attributable to the phenomenon of interest, and how much is attributable to what have come to be known as researcher degrees of freedom (RDF), slows theoretical progress and stymies practical implementation. However, if one could examine the results from a particular set of RDF (known as a universe) against a systematically and comprehensively determined background of alternative viable universes (known as a multiverse), then the effects of RDF can be directly examined to provide greater context and clarity to future researchers, and greater confidence in the recommendations to practitioners. This tutorial demonstrates a means to map result variability directly and efficiently, and empirically investigate RDF impact on conclusions via multiverse analysis. Using the R package multiverse, we outline best practices in planning, executing and interpreting of multiverse analyses.