Affiliation:
1. Department of Basic Psychology and Methodology, University of Murcia, Murcia, Spain
2. Department of Industrial Engineering and Innovation Sciences, Eindhoven University of Technology, Eindhoven, The Netherlands
3. Department of Health Psychology, University of Alicante, San Vicente del Raspeig, Spain
Abstract
Meta-analysis is one of the most useful research approaches, the relevance of which relies on its credibility. Reproducibility of scientific results could be considered as the minimal threshold of this credibility. We assessed the reproducibility of a sample of meta-analyses published between 2000 and 2020. From a random sample of 100 articles reporting results of meta-analyses of interventions in clinical psychology, 217 meta-analyses were selected. We first tried to retrieve the original data by recovering a data file, recoding the data from document files, or requesting it from original authors. Second, through a multistage workflow, we tried to reproduce the main results of each meta-analysis. The original data were retrieved for 67% (146/217) of meta-analyses. Although this rate showed an improvement over the years, in only 5% of these cases was it possible to retrieve a data file ready for reuse. Of these 146, 52 showed a discrepancy larger than 5% in the main results in the first stage. For 10 meta-analyses, this discrepancy was solved after fixing a coding error of our data-retrieval process, and for 15 of them, it was considered approximately reproduced in a qualitative assessment. In the remaining meta-analyses (18%, 27/146), different issues were identified in an in-depth review, such as reporting inconsistencies, lack of data, or transcription errors. Nevertheless, the numerical discrepancies were mostly minor and had little or no impact on the conclusions. Overall, one of the biggest threats to the reproducibility of meta-analysis is related to data availability and current data-sharing practices in meta-analysis.
Funder
ministerio de ciencia e innovación
ministerio de universidades
fundación séneca
Reference38 articles.
1. Allaire J., Xie Y., McPherson J., Luraschi J., Ushey K., Atkins A., Wickham H., Cheng J., Chang W., Iannone R. (2022). Rmarkdown: Dynamic documents for r. https://github.com/rstudio/rmarkdown
2. The reproducibility of statistical results in psychological research: An investigation using unpublished raw data.
3. Aust F., Barth M. (2022). papaja: Prepare reproducible APA journal articles with R Markdown. https://github.com/crsh/papaja
4. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases
5. Chamberlain S., Zhu H., Jahn N., Boettiger C., Ram K. (2023). Rcrossref: Client for various ‘CrossRef’ ‘APIs’ (R package Version 1.2.0) [Computer software]. Retrieved from https://CRAN.R-project.org/package=rcrossref