Many Labs 2: Investigating Variation in Replicability Across Samples and Settings-Reference-Cited by-同舟云学术

Many Labs 2: Investigating Variation in Replicability Across Samples and Settings

Published:2018-12 Issue:4 Volume:1 Page:443-490
ISSN:2515-2459
Container-title:Advances in Methods and Practices in Psychological Science
language:en
Short-container-title:Advances in Methods and Practices in Psychological Science

Author:

Klein Richard A.¹,Vianello Michelangelo²,Hasselman Fred³⁴,Adams Byron G.⁵⁶,Adams Reginald B.⁷,Alper Sinan⁸,Aveyard Mark⁹,Axt Jordan R.¹⁰,Babalola Mayowa T.¹¹,Bahník Štěpán¹²,Batra Rishtee¹³,Berkics Mihály¹⁴,Bernstein Michael J.¹⁵,Berry Daniel R.¹⁶,Bialobrzeska Olga¹⁷,Binan Evans Dami¹⁸,Bocian Konrad¹⁹,Brandt Mark J.⁵,Busching Robert²⁰,Rédei Anna Cabak²¹,Cai Huajian²²,Cambier Fanny²³²⁴,Cantarero Katarzyna²⁵,Carmichael Cheryl L.²⁶,Ceric Francisco²⁷²⁸,Chandler Jesse²⁹³⁰,Chang Jen-Ho³¹³²,Chatard Armand³³³⁴,Chen Eva E.³⁵,Cheong Winnee³⁶,Cicero David C.³⁷,Coen Sharon³⁸,Coleman Jennifer A.³⁹,Collisson Brian⁴⁰,Conway Morgan A.⁴¹,Corker Katherine S.⁴²,Curran Paul G.⁴²,Cushman Fiery⁴³,Dagona Zubairu K.¹⁸,Dalgar Ilker⁴⁴,Dalla Rosa Anna²,Davis William E.⁴⁵,de Bruijn Maaike⁵,De Schutter Leander⁴⁶,Devos Thierry⁴⁷,de Vries Marieke³⁴⁸⁴⁹,Doğulu Canay⁵⁰,Dozo Nerisa⁵¹,Dukes Kristin Nicole⁵²,Dunham Yarrow⁵³,Durrheim Kevin⁵⁴,Ebersole Charles R.⁵⁵,Edlund John E.⁵⁶,Eller Anja⁵⁷,English Alexander Scott⁵⁸,Finck Carolyn⁵⁹,Frankowska Natalia¹⁷,Freyre Miguel-Ángel⁵⁷,Friedman Mike²³²⁴,Galliani Elisa Maria⁶⁰,Gandi Joshua C.¹⁸,Ghoshal Tanuka⁶¹,Giessner Steffen R.⁶²,Gill Tripat⁶³,Gnambs Timo⁶⁴⁶⁵,Gómez Ángel⁶⁶,González Roberto⁶⁷,Graham Jesse⁶⁸,Grahe Jon E.⁶⁹,Grahek Ivan⁷⁰,Green Eva G. T.⁷¹,Hai Kakul⁷²,Haigh Matthew⁷³,Haines Elizabeth L.⁷⁴,Hall Michael P.⁷⁵,Heffernan Marie E.⁷⁶,Hicks Joshua A.⁷⁷,Houdek Petr⁷⁸,Huntsinger Jeffrey R.⁷⁹,Huynh Ho Phi⁸⁰,IJzerman Hans¹,Inbar Yoel⁸¹,Innes-Ker Åse H.⁸²,Jiménez-Leal William⁵⁹,John Melissa-Sue⁸³,Joy-Gaba Jennifer A.³⁹,Kamiloğlu Roza G.⁸⁴,Kappes Heather Barry⁸⁵,Karabati Serdar⁸⁶,Karick Haruna¹⁷¹⁸,Keller Victor N.⁸⁷,Kende Anna⁸⁸,Kervyn Nicolas²³²⁴,Knežević Goran⁸⁹,Kovacs Carrie⁹⁰,Krueger Lacy E.⁹¹,Kurapov German⁹²,Kurtz Jamie⁹³,Lakens Daniël⁹⁴,Lazarević Ljiljana B.⁹⁵,Levitan Carmel A.⁹⁶,Lewis Neil A.⁹⁷,Lins Samuel⁹⁸,Lipsey Nikolette P.⁴¹,Losee Joy E.⁴¹,Maassen Esther⁹⁹,Maitner Angela T.⁹,Malingumu Winfrida¹⁰⁰,Mallett Robyn K.⁷⁹,Marotta Satia A.¹⁰¹,Međedović Janko¹⁰²¹⁰³,Mena-Pacheco Fernando¹⁰⁴,Milfont Taciano L.¹⁰⁵,Morris Wendy L.¹⁰⁶,Murphy Sean C.¹⁰⁷,Myachykov Andriy⁷³,Neave Nick⁷³,Neijenhuijs Koen¹⁰⁸¹⁰⁹,Nelson Anthony J.⁷,Neto Félix⁹⁸,Lee Nichols Austin¹¹⁰,Ocampo Aaron¹⁰⁴,O’Donnell Susan L.¹¹¹,Oikawa Haruka¹¹²,Oikawa Masanori¹¹²,Ong Elsie¹¹³,Orosz Gábor¹¹⁴,Osowiecka Malgorzata¹⁷,Packard Grant⁶³,Pérez-Sánchez Rolando¹¹⁵,Petrović Boban¹⁰³,Pilati Ronaldo⁸⁷,Pinter Brad⁷,Podesta Lysandra³⁴,Pogge Gabrielle⁴¹,Pollmann Monique M. H.¹¹⁶,Rutchick Abraham M.¹¹⁷,Saavedra Patricio¹¹⁸,Saeri Alexander K.¹¹⁹,Salomon Erika¹²⁰,Schmidt Kathleen¹²¹,Schönbrodt Felix D.¹²²,Sekerdej Maciej B.¹²³,Sirlopú David²⁷,Skorinko Jeanine L. M.⁸³,Smith Michael A.⁷³,Smith-Castro Vanessa¹¹⁵,Smolders Karin C. H. J.⁹⁴,Sobkow Agata¹²⁴,Sowden Walter¹²⁵,Spachtholz Philipp¹²²,Srivastava Manini¹²⁶,Steiner Troy G.⁷,Stouten Jeroen¹²⁷,Street Chris N. H.¹²⁸,Sundfelt Oskar K.⁸²,Szeto Stephanie³⁸,Szumowska Ewa¹²³,Tang Andrew C. W.¹¹³,Tanzer Norbert¹²⁹,Tear Morgan J.¹¹⁹,Theriault Jordan¹³⁰,Thomae Manuela¹³¹,Torres David¹³²,Traczyk Jakub¹²⁴,Tybur Joshua M.¹³³,Ujhelyi Adrienn⁸⁸,van Aert Robbie C. M.⁹⁹,van Assen Marcel A. L. M.⁹⁹,van der Hulst Marije¹³⁴,van Lange Paul A. M.¹³³,van ’t Veer Anna Elisabeth¹³⁵,Vásquez- Echeverría Alejandro¹³⁶,Ann Vaughn Leigh¹³⁷,Vázquez Alexandra⁶⁶,Vega Luis Diego¹⁰⁴,Verniers Catherine¹³⁸,Verschoor Mark¹³⁹,Voermans Ingrid P. J.⁴,Vranka Marek A.¹⁴⁰,Welch Cheryl⁹³,Wichman Aaron L.¹⁴¹,Williams Lisa A.¹⁴²,Wood Michael¹³¹,Woodzicka Julie A.¹⁴³,Wronska Marta K.¹⁹,Young Liane¹⁴⁴,Zelenski John M.¹⁴⁵,Zhijia Zeng¹⁴⁶,Nosek Brian A.⁵⁵¹⁴⁷

Affiliation:

1. Laboratoire Inter-universitaire de Psychologie, Personnalité, Cognition, Changement Social (LIP/PC2S), Université Grenoble Alpes

2. Department of Philosophy, Sociology, Education and Applied Psychology, University of Padua

3. Behavioural Science Institute, Radboud University Nijmegen

4. School of Pedagogical and Educational Sciences, Radboud University Nijmegen

5. Department of Social Psychology, Tilburg University

6. Department of Industrial Psychology and People Management, University of Johannesburg

7. Department of Psychology, The Pennsylvania State University

8. Department of Psychology, Yasar University

9. Department of International Studies, American University of Sharjah

10. Center for Advanced Hindsight, Duke University

11. College of Business and Economics, United Arab Emirates University

12. Department of Management, Faculty of Business Administration, University of Economics, Prague

13. Erivan K. Haub School of Business, Saint Joseph’s University

14. Institute of Psychology, ELTE Eötvös Loránd University

15. Psychological and Social Sciences Program, Pennsylvania State University Abington

16. Department of Psychology, California State University San Marcos

17. Warsaw Faculty of Psychology, SWPS University of Social Sciences and Humanities

18. Department of General and Applied Psychology, University of Jos

19. Sopot Faculty of Psychology, SWPS University of Social Sciences and Humanities

20. Department of Psychology, University of Potsdam

21. Centre for Languages and Literature, Lund University

22. Institute of Psychology, Chinese Academy of Sciences

23. Louvain Research Institute in Management and Organizations (LouRIM), Université catholique de Louvain

24. Center on Consumers and Marketing Strategy (CCMS), Université catholique de Louvain

25. Social Behavior Research Centre, Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities

26. Department of Psychology, Brooklyn College & Graduate Center, CUNY

27. Facultad de Psicologia, Universidad del Desarrollo

28. Centro de Apego y Regulacion Emocional, Universidad del Desarrollo

29. Institute for Social Research, University of Michigan

30. Mathematica Policy Research, Princeton, New Jersey

31. Institute of Ethnology, Academia Sinica

32. Department of Psychology, National Taiwan University

33. Department of Psychology, Poitiers University

34. CNRS Unité Mixte de Recherche 7295, Poitiers, France

35. Division of Social Science, The Hong Kong University of Science and Technology

36. Department of Psychology, HELP University

37. Department of Psychology, University of Hawaii at Manoa

38. Directorate of Psychology and Public Health, University of Salford

39. Department of Psychology, Virginia Commonwealth University

40. Department of Psychology, Azusa Pacific University

41. Department of Psychology, University of Florida

42. Department of Psychology, Grand Valley State University

43. Department of Psychology, Harvard University

44. Department of Psychology, Middle East Technical University

45. Department of Psychology, Wittenberg University

46. Leadership and Human Resource Management, WHU – Otto Beisheim School of Management

47. Department of Psychology, San Diego State University

48. Institute for Computing and Information Sciences, Radboud University Nijmegen

49. Tilburg Institute for Behavioral Economics Research, Tilburg University

50. Department of Psychology, Başkent University

51. School of Psychology, The University of Queensland

52. Office of Institutional Diversity, Allegheny College

53. Department of Psychology, Yale University

54. School of Applied Human Sciences, University of KwaZulu-Natal

55. Department of Psychology, University of Virginia

56. Department of Psychology, Rochester Institute of Technology

57. Facultad de Psicología, Universidad Nacional Autónoma de México

58. Shanghai Intercultural Institute, Shanghai International Studies University

59. Departamento de Psicología, Universidad de los Andes, Colombia

60. Department of Political and Juridical Sciences and International Studies, University of Padua

61. Department of Marketing and International Business, Baruch College, CUNY

62. Department of Organisation and Personnel Management, Rotterdam School of Management, Erasmus University

63. Lazaridis School of Business and Economics, Wilfrid Laurier University

64. Educational Measurement, Leibniz Institute for Educational Trajectories, Bamberg, Germany

65. Institute of Education and Psychology, Johannes Kepler University Linz

66. Departamento de Psicología Social y de las Organizaciones, Universidad Nacional de Educación a Distancia

67. Escuela de Psicología, Pontificia Universidad Católica de Chile

68. Eccles School of Business, University of Utah

69. Psychology, Pacific Lutheran University

70. Department of Experimental Clinical and Health Psychology, Ghent University

71. Institute of Psychology, Faculty of Social and Political Sciences, University of Lausanne

72. Amity Institute of Psychology and Allied Sciences, Amity University

73. Department of Psychology, Northumbria University

74. Department of Psychology, William Paterson University

75. Department of Psychology, University of Michigan

76. Smith Child Health Research, Outreach, and Advocacy Center, Ann & Robert H. Lurie Children’s Hospital of Chicago, Chicago, Illinois

77. Department of Psychological & Brain Sciences, Texas A&M University

78. Department of Economics and Management, Faculty of Social and Economic Studies, Jan Evangelista Purkyne University

79. Department of Psychology, Loyola University Chicago

80. Department of Science and Mathematics, Texas A&M University-San Antonio

81. Department of Psychology, University of Toronto Scarborough

82. Department of Psychology, Lund University

83. Department of Social Science and Policy Studies, Worcester Polytechnic Institute

84. Department of Psychology, University of Amsterdam

85. Department of Management, London School of Economics and Political Science

86. Department of Business Administration, Istanbul Bilgi University

87. Department of Social and Work Psychology, Institute of Psychology, University of Brasilia

88. Department of Social Psychology, ELTE Eötvös Loránd University

89. Department of Psychology, Faculty of Philosophy, University of Belgrade

90. Department of Work, Organizational and Media Psychology, Johannes Kepler University Linz

91. Department of Psychology & Special Education, Texas A&M University-Commerce

92. International Victimology Institute Tilburg, Tilburg University

93. Department of Psychology, James Madison University

94. School of Innovation Science, Eindhoven University of Technology

95. Institute of Psychology, Faculty of Philosophy, University of Belgrade

96. Department of Cognitive Science, Occidental College

97. Department of Communication, Cornell University

98. Department of Psychology, University of Porto

99. Department of Methodology and Statistics, Tilburg University

100. Department of Education Policy Planning and Administration, Faculty of Education, Open University of Tanzania

101. Department of Occupational Therapy, Tufts University

102. Faculty of Media and Communications, Singidunum University

103. Institute of Criminological and Sociological Research, Belgrade, Serbia

104. Department of Psychology, Universidad Latina de Costa Rica

105. Centre for Applied Cross-Cultural Research, Victoria University of Wellington

106. Department of Psychology, McDaniel College

107. Melbourne School of Psychological Sciences, The University of Melbourne

108. Department of Clinical, Neuro- and Developmental Psychology, Vrije Universiteit Amsterdam

109. Amsterdam Public Health Research Institute, Amsterdam, The Netherlands

110. Department of Psychology, University of Central Florida

111. Department of Psychology, George Fox University

112. Department of Psychology, Doshisha University

113. Li Ka Shing Institute of Professional and Continuing Education (LiPACE), The Open University of Hong Kong

114. Department of Psychology, Stanford University

115. Institute for Psychological Research, University of Costa Rica

116. Department of Communication and Cognition, Tilburg University

117. Department of Psychology, California State University, Northridge

118. School of Psychology, University of Sussex

119. BehaviourWorks Australia, Monash Sustainable Development Institute, Monash University

120. Department of Computer Science, University of Chicago

121. Department of Psychology, Southern Illinois University Carbondale

122. Department of Psychology, Ludwig-Maximilians-Universität München

123. Institute of Psychology, Jagiellonian University in Kraków

124. Wroclaw Faculty of Psychology, SWPS University of Social Sciences and Humanities

125. Center for Military Psychiatry and Neuroscience, Walter Reed Army Institute of Research, Silver Spring, Maryland

126. Faculty of Psychology and Educational Science and Physical Education, University of Regensburg

127. Occupational & Organisational Psychology and Professional Learning, KU Leuven

128. Department of Psychology, University of Huddersfield

129. Institute of Psychology, University of Graz

130. Department of Psychology, Northeastern University

131. Department of Psychology, University of Winchester

132. Department of Psychology, Universidad de Iberoamerica

133. Department of Experimental and Applied Psychology, Vrije Universiteit Amsterdam

134. Department of Obstetrics and Gynaecology, Erasmus MC, Rotterdam, The Netherlands

135. Methodology and Statistics Unit, Institute of Psychology, Leiden University

136. Centro de Investigación Básica en Psicología, Universidad de la República

137. Department of Psychology, Ithaca College

138. Institute of Psychology, Paris Descartes University - Sorbonne Paris Cité

139. Department of Social Psychology, University of Groningen

140. Department of Psychology, Faculty of Arts, Charles University

141. Department of Psychological Science, Western Kentucky University

142. School of Psychology, University of New South Wales

143. Department of Psychology, Washington and Lee University

144. Department of Psychology, Boston College

145. Department of Psychology, Carleton University

146. Zhejiang University of Finance and Economics

147. Center for Open Science, Charlottesville, Virginia

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance ( p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion ( p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.

Funder

Association for Psychological Science

Publisher

SAGE Publications

Link

http://journals.sagepub.com/doi/pdf/10.1177/2515245918810225

Reference110 articles.

1. Socioeconomic status and health: The challenge of the gradient.

2. Relationship of subjective and objective social status with psychological and physiological functioning: Preliminary data in healthy, White women.

3. Overcoming intuition: Metacognitive difficulty activates analytic reasoning.

4. The Local-Ladder Effect

5. Who I Am Depends on How I Feel

Cited by 596 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Replicable and robust cellular and biochemical blood marker signatures of depression and depressive symptoms;Psychiatry Research;2024-12

2. Revisiting the moral forecasting error – A preregistered replication and extension of “Are we more moral than we think?”;Journal of Experimental Social Psychology;2024-11

3. The ways of the world? Cross-sample replicability of personality trait-life outcome associations;Journal of Research in Personality;2024-10

4. The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche;Frontiers in Psychology;2024-09-12

5. The opportunities and limits of open science for LGBTIQ+ research;Journal of Social Issues;2024-09-11