Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability-Reference-Cited by-同舟云学术

Many Labs 5: Testing Pre-Data-Collection Peer Review as an Intervention to Increase Replicability

Published:2020-09 Issue:3 Volume:3 Page:309-331
ISSN:2515-2459
Container-title:Advances in Methods and Practices in Psychological Science
language:en
Short-container-title:Advances in Methods and Practices in Psychological Science

Author:

Ebersole Charles R.¹^ORCID,Mathur Maya B.²,Baranski Erica³,Bart-Plange Diane-Jo¹,Buttrick Nicholas R.¹,Chartier Christopher R.⁴,Corker Katherine S.⁵,Corley Martin⁶,Hartshorne Joshua K.⁷,IJzerman Hans⁸⁹,Lazarević Ljiljana B.¹⁰¹¹,Rabagliati Hugh⁶,Ropovik Ivan¹²¹³,Aczel Balazs¹⁴,Aeschbach Lena F.¹⁵,Andrighetto Luca¹⁶,Arnal Jack D.¹⁷,Arrow Holly¹⁸,Babincak Peter¹⁹,Bakos Bence E.¹⁴,Baník Gabriel¹⁹,Baskin Ernest²⁰,Belopavlović Radomir²¹,Bernstein Michael H.²²²³,Białek Michał²⁴,Bloxsom Nicholas G.⁴,Bodroža Bojana²¹,Bonfiglio Diane B. V.⁴,Boucher Leanne²⁵,Brühlmann Florian¹⁵,Brumbaugh Claudia C.²⁶,Casini Erica²⁷,Chen Yiling²⁸,Chiorri Carlo¹⁶,Chopik William J.²⁹,Christ Oliver³⁰,Ciunci Antonia M.²³,Claypool Heather M.³¹,Coary Sean³²,Čolić Marija V.³³,Collins W. Matthew²⁵,Curran Paul G.⁵,Day Chris R.³⁴,Dering Benjamin³⁵,Dreber Anna³⁶³⁷,Edlund John E.³⁸,Falcão Filipe³⁹,Fedor Anna⁴⁰,Feinberg Lily⁷,Ferguson Ian R.⁴¹⁴²,Ford Máire⁴³,Frank Michael C.⁴⁴,Fryberger Emily⁴⁵,Garinther Alexander¹⁸,Gawryluk Katarzyna⁴⁶,Ashbaugh Kayla⁴⁷,Giacomantonio Mauro⁴⁸,Giessner Steffen R.⁴⁹,Grahe Jon E.⁴⁵,Guadagno Rosanna E.⁵⁰,Hałasa Ewa⁵¹,Hancock Peter J. B.³⁵,Hilliard Rias A.⁵²,Hüffmeier Joachim⁵³,Hughes Sean⁵⁴,Idzikowska Katarzyna⁴⁶,Inzlicht Michael⁵⁵,Jern Alan⁵²,Jiménez-Leal William⁵⁶,Johannesson Magnus³⁶,Joy-Gaba Jennifer A.⁴¹,Kauff Mathias⁵⁷,Kellier Danielle J.⁵⁸,Kessinger Grecia⁵⁹,Kidwell Mallory C.⁶⁰,Kimbrough Amanda M.⁶¹,King Josiah P. J.⁶,Kolb Vanessa S.²³,Kołodziej Sabina⁴⁶,Kovacs Marton¹⁴,Krasuska Karolina⁵¹,Kraus Sue⁶²,Krueger Lacy E.⁶³,Kuchno Katarzyna⁵¹,Lage Caio Ambrosio⁶⁴,Langford Eleanor V.¹,Levitan Carmel A.⁶⁵,de Lima Tiago Jessé Souza⁶⁶,Lin Hause⁵⁵,Lins Samuel³⁹,Loy Jia E.⁶⁷,Manfredi Dylan⁶⁸,Markiewicz Łukasz⁴⁶,Menon Madhavi²⁵,Mercier Brett⁶⁹,Metzger Mitchell⁴,Meyet Venus⁵⁹,Millen Ailsa E.³⁵,Miller Jeremy K.⁷⁰,Montealegre Andres⁷¹,Moore Don A.⁷²,Muda Rafał⁵¹,Nave Gideon⁶⁸,Nichols Austin Lee⁷³,Novak Sarah A.⁷⁴,Nunnally Christian⁷⁵,Orlić Ana³³,Palinkas Anna¹⁴,Panno Angelo⁷⁶,Parks Kimberly P.¹,Pedović Ivana⁷⁷,Pękala Emilian⁵¹,Penner Matthew R.⁷⁸,Pessers Sebastiaan⁷⁹,Petrović Boban¹¹⁸⁰,Pfeiffer Thomas⁸¹,Pieńkosz Damian⁵¹,Preti Emanuele²⁷,Purić Danka¹¹⁸²,Ramos Tiago³⁹,Ravid Jonathan⁷,Razza Timothy S.²⁵,Rentzsch Katrin⁸³,Richetin Juliette²⁷,Rife Sean C.⁸⁴,Rosa Anna Dalla⁸⁵,Rudy Kaylis Hase⁵⁹,Salamon Janos¹⁴⁸⁶,Saunders Blair⁸⁷,Sawicki Przemysław⁴⁶,Schmidt Kathleen⁸⁸,Schuepfer Kurt³¹,Schultze Thomas⁸⁹⁹⁰,Schulz-Hardt Stefan⁸⁹⁹⁰,Schütz Astrid⁹¹,Shabazian Ani N.⁹²,Shubella Rachel L.⁵²,Siegel Adam⁹³,Silva Rúben³⁹,Sioma Barbara⁵¹,Skorb Lauren⁷,de Souza Luana Elayne Cunha⁹⁴,Steegen Sara⁷⁹,Stein L. A. R.²²²³⁹⁵,Sternglanz R. Weylin²⁵,Stojilović Darko⁹⁶,Storage Daniel⁹⁷,Sullivan Gavin Brent³⁴,Szaszi Barnabas¹⁴,Szecsi Peter¹⁴,Szöke Orsolya¹⁴,Szuts Attila¹⁴,Thomae Manuela⁹⁸⁹⁹,Tidwell Natasha D.⁶²,Tocco Carly²⁶,Torka Ann-Kathrin⁵³,Tuerlinckx Francis⁷⁹,Vanpaemel Wolf⁷⁹,Vaughn Leigh Ann¹⁰⁰,Vianello Michelangelo⁸⁵,Viganola Domenico³⁶,Vlachou Maria⁷⁹,Walker Ryan J.³¹,Weissgerber Sophia C.¹⁰¹,Wichman Aaron L.⁷⁸,Wiggins Bradford J.⁵⁹,Wolf Daniel⁹¹,Wood Michael J.¹⁰²,Zealley David⁵⁹,Žeželj Iris¹¹⁸²,Zrubka Mark¹⁰³,Nosek Brian A.¹¹⁰⁴

Affiliation:

1. Department of Psychology, University of Virginia

2. Quantitative Sciences Unit, Stanford University

3. Department of Psychology, The University of Houston

4. Department of Psychology, Ashland University

5. Department of Psychology, Grand Valley State University

6. Psychology, School of Philosophy, Psychology & Language Sciences, University of Edinburgh

7. Psychology and Neuroscience Department, Boston College

8. LIP/PC2S, Université Grenoble Alpes

9. Institut Universitaire de France

10. Institute of Psychology, Faculty of Philosophy, University of Belgrade

11. Laboratory for Research of Individual Differences, Faculty of Philosophy, University of Belgrade

12. Institute for Research and Development of Education, Faculty of Education, Charles University

13. Faculty of Education, University of Presov

14. Institute of Psychology, ELTE Eötvös Loránd University

15. Department of Psychology, University of Basel

16. Department of Educational Science, University of Genova

17. Psychology Department, McDaniel College

18. Department of Psychology, University of Oregon

19. Institute of Psychology, Faculty of Arts, University of Presov

20. Department of Food Marketing, Haub School of Business, Saint Joseph’s University

21. Department of Psychology, Faculty of Philosophy, University of Novi Sad

22. Department of Behavioral and Social Sciences, School of Public Health, Brown University

23. Department of Psychology, University of Rhode Island

24. Institute of Psychology, University of Wroclaw

25. Department of Psychology and Neuroscience, Nova Southeastern University

26. Department of Psychology, Queens College, City University of New York

27. Department of Psychology, University of Milano-Bicocca

28. John A. Paulson School of Engineering and Applied Sciences, Harvard University

29. Department of Psychology, Michigan State University

30. Faculty of Psychology, FernUniversität in Hagen

31. Department of Psychology, Miami University

32. Quinlan School of Business, Loyola University Chicago

33. Faculty of Sport and Physical Education, University of Belgrade

34. Centre for Trust, Peace and Social Relations, Coventry University

35. Psychology, University of Stirling

36. Department of Economics, Stockholm School of Economics

37. Department of Economics, University of Innsbruck

38. Department of Psychology, Rochester Institute of Technology

39. Department of Psychology, University of Porto

40. Institute of Evolution, Centre for Ecological Research, Budapest, Hungary

41. Department of Psychology, Virginia Commonwealth University

42. Department of Psychology, New York University

43. Department of Psychology, Loyola Marymount University

44. Department of Psychology, Stanford University

45. Department of Psychology, Pacific Lutheran University

46. Department of Economic Psychology, Kozminski University

47. Department of Biology and Biomedical Engineering, Rose-Hulman Institute of Technology

48. Department of Social & Developmental Psychology, Sapienza University of Rome

49. Rotterdam School of Management, Erasmus University

50. Center for International Security and Cooperation, Stanford University

51. Faculty of Economics, Maria Curie-Skłodowska University

52. Department of Humanities, Social Sciences, and the Arts, Rose-Hulman Institute of Technology

53. Department of Psychology, TU Dortmund University

54. Department of Experimental Clinical and Health Psychology, Ghent University

55. Department of Psychology, University of Toronto

56. Department of Psychology, Universidad de los Andes

57. Department of Psychology, Medical School Hamburg

58. Perelman School of Medicine, University of Pennsylvania

59. Department of Psychology, Brigham Young University–Idaho

60. Department of Psychology, University of Utah

61. School of Arts, Technology, Emerging Media, & Communication, University of Texas at Dallas

62. Psychology, Fort Lewis College

63. Department of Psychology & Special Education, Texas A&M University-Commerce

64. Department of Psychology, Pontifical Catholic University of Rio de Janeiro

65. Department of Cognitive Science, Occidental College

66. Department of Social and Work Psychology, University of Brasília

67. Linguistics & English Language, School of Philosophy, Psychology & Language Sciences, University of Edinburgh

68. Marketing Department, The Wharton School of the University of Pennsylvania

69. Department of Psychological Science, University of California, Irvine

70. Department of Psychology, Willamette University

71. Department of Psychology, Cornell University

72. Haas School of Business, University of California at Berkeley

73. Institute of Social and Economic Research, Duy Tan University

74. Department of Psychology, Hofstra University

75. Department of Computer Science and Software Engineering, Rose-Hulman Institute of Technology

76. Department of Human Science, European University of Rome

77. Department of Psychology, Faculty of Philosophy, University of Niš

78. Department of Psychological Sciences, Western Kentucky University

79. Psychology and Educational Sciences, University of Leuven

80. Institute of Criminological and Sociological Research, Belgrade, Serbia

81. New Zealand Institute for Advanced Study, Massey University

82. Department of Psychology, Faculty of Philosophy, University of Belgrade

83. Psychologische Hochschule Berlin

84. Department of Psychology, Murray State University

85. Department of Philosophy, Sociology, Education and Applied Psychology, University of Padova

86. Doctoral School of Psychology, ELTE Eötvös Loránd University

87. School of Social Sciences, University of Dundee

88. School of Psychological and Behavioral Sciences, Southern Illinois University Carbondale

89. Institute of Psychology, University of Göttingen

90. Leibniz Science Campus Primate Cognition, Göttingen, Germany

91. Department of Psychology, University of Bamberg

92. School of Education, Loyola Marymount University

93. Cultivate Labs, Chicago, Illinois

94. Department of Psychology, University of Fortaleza

95. Rhode Island Training School, Rhode Island Department of Children, Youth and Families

96. Belgrade, Serbia

97. Department of Psychology, University of Denver

98. MEU - Die Multiversität

99. Diploma University of Applied Sciences

100. Department of Psychology, Ithaca College

101. Department of Psychology, University of Kassel

102. Department of Psychology, University of Winchester

103. Department of Psychology, University of Amsterdam

104. Center for Open Science, Charlottesville, Virginia

Abstract

Replication studies in psychological science sometimes fail to reproduce prior findings. If these studies use methods that are unfaithful to the original study or ineffective in eliciting the phenomenon of interest, then a failure to replicate may be a failure of the protocol rather than a challenge to the original finding. Formal pre-data-collection peer review by experts may address shortcomings and increase replicability rates. We selected 10 replication studies from the Reproducibility Project: Psychology (RP:P; Open Science Collaboration, 2015) for which the original authors had expressed concerns about the replication designs before data collection; only one of these studies had yielded a statistically significant effect ( p < .05). Commenters suggested that lack of adherence to expert review and low-powered tests were the reasons that most of these RP:P studies failed to replicate the original effects. We revised the replication protocols and received formal peer review prior to conducting new replication studies. We administered the RP:P and revised protocols in multiple laboratories (median number of laboratories per original study = 6.5, range = 3–9; median total sample = 1,279.5, range = 276–3,512) for high-powered tests of each original finding with both protocols. Overall, following the preregistered analysis plan, we found that the revised protocols produced effect sizes similar to those of the RP:P protocols (Δ r = .002 or .014, depending on analytic approach). The median effect size for the revised protocols ( r = .05) was similar to that of the RP:P protocols ( r = .04) and the original RP:P replications ( r = .11), and smaller than that of the original studies ( r = .37). Analysis of the cumulative evidence across the original studies and the corresponding three replication attempts provided very precise estimates of the 10 tested effects and indicated that their effect sizes (median r = .07, range = .00–.15) were 78% smaller, on average, than the original effect sizes (median r = .37, range = .19–.50).

Funder

Laura and John Arnold Foundation

Association for Psychological Science

Publisher

SAGE Publications

Subject

General Psychology

Link

http://journals.sagepub.com/doi/pdf/10.1177/2515245920958687

Reference59 articles.

1. Increasing and decreasing motor and cognitive output: A model of general action and inaction goals.

2. Response to Comment on “Estimating the reproducibility of psychological science”