Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty-Reference-Cited by-同舟云学术

Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty

Published:2022-10-28 Issue:44 Volume:119 Page:
ISSN:0027-8424
Container-title:Proceedings of the National Academy of Sciences
language:en
Short-container-title:Proc. Natl. Acad. Sci. U.S.A.

Author:

Breznau Nate¹^ORCID,Rinke Eike Mark²^ORCID,Wuttke Alexander³⁴^ORCID,Nguyen Hung H. V.¹⁵^ORCID,Adem Muna⁶,Adriaans Jule⁷^ORCID,Alvarez-Benjumea Amalia⁸^ORCID,Andersen Henrik K.⁹^ORCID,Auer Daniel³^ORCID,Azevedo Flavio¹⁰^ORCID,Bahnsen Oke¹¹^ORCID,Balzer Dave¹²^ORCID,Bauer Gerrit¹³^ORCID,Bauer Paul C.³^ORCID,Baumann Markus¹⁴¹⁵,Baute Sharon¹⁶,Benoit Verena⁴¹⁷^ORCID,Bernauer Julian³^ORCID,Berning Carl¹⁸,Berthold Anna¹⁷^ORCID,Bethke Felix S.¹⁹^ORCID,Biegert Thomas²⁰^ORCID,Blinzler Katharina²¹,Blumenberg Johannes N.²²,Bobzien Licia²³,Bohman Andrea²⁴^ORCID,Bol Thijs²⁵²⁶^ORCID,Bostic Amie²⁷^ORCID,Brzozowska Zuzanna²⁸²⁹,Burgdorf Katharina¹¹,Burger Kaspar²⁵³⁰³¹^ORCID,Busch Kathrin B.³²^ORCID,Carlos-Castillo Juan³³³⁴^ORCID,Chan Nathan³⁵,Christmann Pablo³⁶^ORCID,Connelly Roxanne³⁷^ORCID,Czymara Christian S.³⁸^ORCID,Damian Elena³⁹^ORCID,Ecker Alejandro³,Edelmann Achim⁴⁰^ORCID,Eger Maureen A.²⁴^ORCID,Ellerbrock Simon³¹¹^ORCID,Forke Anna³²,Forster Andrea⁴¹^ORCID,Gaasendam Chris⁴²,Gavras Konstantin¹¹^ORCID,Gayle Vernon³⁷^ORCID,Gessler Theresa⁴³^ORCID,Gnambs Timo⁴⁴^ORCID,Godefroidt Amélie⁴⁵^ORCID,Grömping Max⁴⁶^ORCID,Groß Martin⁴⁷,Gruber Stefan⁴⁸^ORCID,Gummer Tobias³⁶^ORCID,Hadjar Andreas⁴⁹⁵⁰⁵¹⁵²^ORCID,Heisig Jan Paul⁵³⁵⁴^ORCID,Hellmeier Sebastian⁵⁵^ORCID,Heyne Stefanie³^ORCID,Hirsch Magdalena⁵⁶^ORCID,Hjerm Mikael²⁴,Hochman Oshrat³⁶^ORCID,Hövermann Andreas⁵⁰⁵⁷^ORCID,Hunger Sophia⁵⁸^ORCID,Hunkler Christian⁵⁹^ORCID,Huth Nora⁶⁰^ORCID,Ignácz Zsófia S.³⁸^ORCID,Jacobs Laura⁶¹^ORCID,Jacobsen Jannes⁶²⁶³^ORCID,Jaeger Bastian⁶⁴^ORCID,Jungkunz Sebastian⁶⁵⁶⁶⁶⁷^ORCID,Jungmann Nils²¹^ORCID,Kauff Mathias⁶⁸^ORCID,Kleinert Manuel⁶⁹,Klinger Julia⁷⁰^ORCID,Kolb Jan-Philipp⁷¹^ORCID,Kołczyńska Marta⁷²^ORCID,Kuk John⁷³,Kunißen Katharina¹²^ORCID,Kurti Sinatra Dafina³²^ORCID,Langenkamp Alexander³⁸^ORCID,Lersch Philipp M.⁷⁷⁴^ORCID,Löbel Lea-Maria⁷^ORCID,Lutscher Philipp⁷⁵^ORCID,Mader Matthias⁷⁶^ORCID,Madia Joan E.⁷⁷⁷⁸^ORCID,Malancu Natalia⁷⁹,Maldonado Luis⁸⁰^ORCID,Marahrens Helge⁶^ORCID,Martin Nicole⁸¹,Martinez Paul⁸²^ORCID,Mayerl Jochen⁹^ORCID,Mayorga Oscar J.⁸³^ORCID,McManus Patricia⁶^ORCID,McWagner Kyle⁸⁴,Meeusen Cecil⁴²,Meierrieks Daniel⁵⁶^ORCID,Mellon Jonathan⁸¹^ORCID,Merhout Friedolin⁸⁵^ORCID,Merk Samuel⁸⁶^ORCID,Meyer Daniel⁸⁷^ORCID,Micheli Leticia⁸⁸^ORCID,Mijs Jonathan⁸⁹^ORCID,Moya Cristóbal⁹⁰^ORCID,Neunhoeffer Marcel¹¹,Nüst Daniel⁹¹^ORCID,Nygård Olav⁹²^ORCID,Ochsenfeld Fabian⁹³,Otte Gunnar¹²^ORCID,Pechenkina Anna O.⁹⁴^ORCID,Prosser Christopher⁹⁵^ORCID,Raes Louis⁹⁶^ORCID,Ralston Kevin³⁷^ORCID,Ramos Miguel R.⁹⁷,Roets Arne⁹⁸^ORCID,Rogers Jonathan⁹⁹^ORCID,Ropers Guido¹¹^ORCID,Samuel Robin⁴⁹⁵²^ORCID,Sand Gregor⁴⁸^ORCID,Schachter Ariela¹⁰⁰^ORCID,Schaeffer Merlin¹⁰¹^ORCID,Schieferdecker David¹⁰²^ORCID,Schlueter Elmar⁶⁹^ORCID,Schmidt Regine¹⁷,Schmidt Katja M.⁷^ORCID,Schmidt-Catran Alexander³⁸^ORCID,Schmiedeberg Claudia¹³^ORCID,Schneider Jürgen¹⁰³^ORCID,Schoonvelde Martijn¹⁰⁴¹⁰⁵^ORCID,Schulte-Cloos Julia¹⁰⁶,Schumann Sandy¹⁰⁷^ORCID,Schunck Reinhard⁶⁰^ORCID,Schupp Jürgen⁷^ORCID,Seuring Julian¹⁰⁸^ORCID,Silber Henning¹⁰⁹^ORCID,Sleegers Willem⁶⁴^ORCID,Sonntag Nico¹²^ORCID,Staudt Alexander³²,Steiber Nadia¹¹⁰^ORCID,Steiner Nils¹⁸^ORCID,Sternberg Sebastian³²,Stiers Dieter¹¹¹^ORCID,Stojmenovska Dragana²⁶^ORCID,Storz Nora¹¹²^ORCID,Striessnig Erich¹¹³^ORCID,Stroppe Anne-Kathrin²¹^ORCID,Teltemann Janna¹¹⁴^ORCID,Tibajev Andrey⁹²^ORCID,Tung Brian¹⁰⁰^ORCID,Vagni Giacomo²⁵,Van Assche Jasper⁹⁸¹¹⁵^ORCID,van der Linden Meta¹¹²,van der Noll Jolanda¹¹⁶^ORCID,Van Hootegem Arno⁴²^ORCID,Vogtenhuber Stefan¹¹⁷^ORCID,Voicu Bogdan¹¹⁸¹¹⁹^ORCID,Wagemans Fieke¹²⁰¹²¹,Wehl Nadja¹²²,Werner Hannah¹¹¹,Wiernik Brenton M.¹²³,Winter Fabian⁸^ORCID,Wolf Christof³¹¹¹²⁴^ORCID,Yamada Yuki¹²⁵^ORCID,Zhang Nan³,Ziller Conrad⁶⁵¹²⁶^ORCID,Zins Stefan¹²⁷,Żółtak Tomasz⁷²^ORCID

Affiliation:

1. Research Center on Inequality and Social Policy (SOCIUM), University of Bremen, Bremen, 28359, Germany

2. School of Politics and International Studies, University of Leeds, Leeds, LS2 9JT, United Kingdom

3. Mannheim Centre for European Social Research, University of Mannheim, 68131 Mannheim, Germany

4. Department of Political Science, Ludwig Maximilian University, 80539 Munich, Germany

5. Bremen International Graduate School of Social Sciences, 28359 Bremen, Germany

6. Department of Sociology, Indiana University, Bloomington, IN 47405

7. Socio-Economic Panel Study (SOEP), German Institute for Economic Research (DIW), 10117 Berlin, Germany

8. Mechanisms of Normative Change, Max Planck Institute for Research on Collective Goods, 53113 Bonn, Germany

9. Institute of Sociology, Chemnitz University of Technology, 09126 Chemnitz, Germany

10. Department of Psychology, University of Cambridge, Cambridge, CB23RQ, United Kingdom

11. School of Social Sciences, University of Mannheim, 68159 Mannheim, Germany

12. Institute of Sociology, Johannes Gutenberg University Mainz, 55128 Mainz, Germany

13. Department of Sociology, Ludwig Maximilian University, 80801 Munich, Germany

14. Heidelberg University, 69117 Heidelberg, Germany

15. Institute for Political Science, Goethe University Frankfurt, 60323 Frankfurt, Germany

16. Comparative Political Economy, University of Konstanz, 78457 Konstanz, Germany

17. Faculty of Social Sciences, Economics, and Business Administration, University of Bamberg, 96052 Bamberg, Germany

18. Institute for Political Science, Johannes Gutenberg University Mainz, 55099 Mainz, Germany

19. Research Department on Intrastate Conflict, Peace Research Institute Frankfurt, 60329 Frankfurt, Germany

20. Department of Social Policy, London School of Economics and Political Science, London, WC2A 2AE, United Kingdom

21. Survey Data Curation, Leibniz Institute for the Social Sciences (GESIS), 50667 Cologne, Germany

22. Knowledge Exchange and Outreach, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany

23. Jacques Delors Centre, Hertie School, 10117 Berlin, Germany

24. Department of Sociology, Umeå University, 90187 Umeå, Sweden

25. Social Research Institute, Institute of Education, University College London, London, WC1H 0AL, United Kingdom

26. Department of Sociology, University of Amsterdam, 1001 Amsterdam, The Netherlands

27. Department of Sociology, The University of Texas Rio Grande Valley, Brownsville, TX 78520

28. Vienna Institute of Demography, Austrian Academy of Sciences, 1030 Vienna, Austria

29. Austrian National Public Health Institute, Gesundheit Österreich (GÖG), 1030 Vienna, Austria

30. Department of Sociology, University of Zurich, 8050 Zurich, Switzerland

31. Jacobs Center for Productive Youth, University of Zurich, 8050 Zurich, Switzerland

32. Independent researcher

33. Department of Sociology, University of Chile, Santiago, 7800284, Chile

34. Center for Social Conflict and Cohesion Studies (COES), Pontificia Universidad Católica de Chile, Santiago, 8331150, Chile

35. Department of Political Science and International Relations, Loyola Marymount University, Los Angeles, CA 90045

36. Data and Research on Society, Leibniz Institute for the Social Sciences, 68159 Mannheim, Germany

37. School of Social and Political Science, University of Edinburgh, Edinburgh, EH8 9LD, United Kingdom

38. Institute of Sociology, Goethe University Frankfurt, 60323 Frankfurt, Germany

39. Lifestyle and Chronic Diseases, Epidemiology and Public Health, Sciensano, 1000 Brussels, Belgium

40. Médialab, Sciences Po, 75007 Paris, France

41. Empirical Educational and Higher Education Research, Freie Universität Berlin, 14195 Berlin, Germany

42. Department of Sociology, Center for Sociological Research, KU Leuven, 3000 Leuven, Belgium

43. Kulturwissenschaftliche Fakultät, European University Viadrina, 15230 Frankfurt (Oder), Germany

44. Educational Measurement, Leibniz Institute for Educational Trajectories, 96047 Bamberg, Germany

45. Centre for Research on Peace and Development, KU Leuven, 3000 Leuven, Belgium

46. School of Government and International Relations, Griffith University, Nathan, QLD, 4111, Australia

47. Department of Sociology, University of Tübingen, 72074 Tübingen, Germany

48. Max Planck Institute for Social Law and Social Policy, 80799 Munich, Germany

49. University of Luxembourg, 4365 Esch-sur-Alzette, Luxembourg

50. Wirtschafts- und Sozialwissenschaftliches Institut (WSI), Hans Böckler Foundation, 40474 Düsseldorf, Germany

51. University of Fribourg, 1700 Fribourg, Switzerland

52. Department of Social Sciences, University of Luxembourg, 4366 Esch-sur-Alzette, Luxembourg

53. University of Groningen, 9712 CP Groningen,The Netherlands

54. Research Group "Health and Social Inequality", Berlin Social Science Center (WZB), 10785 Berlin, Germany

55. Transformations of Democracy Unit, Berlin Social Science Center (WZB), 10785 Berlin, Germany

56. Research Unit Migration, Integration, Transnationalization, Berlin Social Science Center (WZB), 10785 Berlin, Germany

57. German Socio-Economic Panel Survey, 10117 Berlin, Germany

58. Center for Civil Society Research, Berlin Social Science Center, 10785 Berlin, Germany

59. Berlin Institute for Integration and Migration Research (BIM), Humboldt University Berlin, 10099 Berlin, Germany

60. School of Human and Social Sciences, University of Wuppertal, 42119 Wuppertal, Germany

61. Department of Political Science, Université Libre de Bruxelles, 1050 Bruxelles, Belgium

62. Zeppelin University, 88045 Friedrichshafen, Germany

63. Cluster "Data-Methods-Monitoring", German Center for Integration and Migration Research (DeZIM),10117 Berlin, Germany

64. Department of Social Psychology, Tilburg University, 5037AB Tilburg, The Netherlands

65. Institute for Socio-Economics, University of Duisburg-Essen, 47057 Duisburg, Germany

66. Institute of Political Science, University of Münster, 48149 Münster, Germany

67. Chair of Political Sociology, University of Bamberg, 96052 Bamberg, Germany

68. Department of Psychology, Medical School Hamburg, 20457 Hamburg, Germany

69. Institute of Sociology, Justus Liebig University of Giessen, 35394 Giessen, Germany

70. Institute of Sociology and Social Psychology, University of Cologne, 50931 Cologne, Germany

71. Federal Statistics Office Germany, Destatis, 65189 Wiesbaden, Germany

72. Department of Research on Social and Institutional Transformations, Institute of Political Studies of the Polish Academy of Sciences, 00-625 Warsaw, Poland

73. Department of Political Science, University of Oklahoma, Norman, OK 73019

74. Department of Social Sciences, Humboldt University Berlin, 10099 Berlin, Germany

75. Department of Political Science, University of Oslo, 0851 Oslo, Norway

76. Department of Politics and Public Administration, University of Konstanz, 78457 Konstanz, Germany

77. Department of Sociology, Nuffield College, University of Oxford, Oxford, OX1 1JD, United Kingdom

78. Institute for the Evaluation of Public Policies, Fondazione Bruno Kessler, 38122 Trento, Italy

79. The Institute of Citizenship Studies (InCite), University of Geneva, 1205 Geneva, Switzerland

80. Instituto de Sociologia, Pontifical Catholic University of Chile, Santiago, 7820436, Chile

81. Department of Politics, University of Manchester, Manchester, M19 2JS, United Kingdom

82. Department of Institutional Research, Western Governors University, Salt Lake City, UT 84107

83. Department of Sociology, University of California, Los Angeles, CA 90095

84. Department of Political Science, The University of California, Irvine, CA 92617

85. Department of Sociology and Centre for Social Data Science, University of Copenhagen, 1353 Copenhagen, Denmark

86. Department of School Development, University of Education Karlsruhe, 76133 Karlsruhe, Germany

87. Department of Education and Social Sciences, University of Cologne, 50931 Cologne, Germany

88. Department of Psychology III, Julius-Maximilians University Würzburg, 97070 Würzburg, Germany

89. Department of Sociology, Boston University, Boston, MA 02215

90. Faculty of Sociology, Bielefeld University, 33615 Bielefeld, Germany

91. Department of Geosciences, University of Münster, 49149 Münster, Germany

92. Division of Migration, Ethnicity and Society (REMESO), Linköping University, 60174 Linköping, Sweden

93. Administrative Headquarters, Max Planck Society, 80539 Berlin, Germany

94. Department of Political Science, Utah State University, Logan, UT 84321

95. Department of Politics, International Relations and Philosophy, Royal Holloway University of London, London, TW20 0EX, United Kingdom

96. Department of Economics, Tilburg University, 5037AB Tilburg, The Netherlands

97. Department of Social Policy, Sociology and Criminology, University of Birmingham, Birmingham, B15 2TT, United Kingdom

98. Department of Developmental, Personality and Social Psychology, Ghent University, 9000 Ghent, Belgium

99. Division of Social Science, New York University Abu Dhabi, Abu Dhabi, 10276, United Arab Emirates

100. Department of Sociology, Washington University in St. Louis, St. Louis, MO 63130

101. Department of Sociology, University of Copenhagen, 1353 Copenhagen, Denmark

102. Institute for Media and Communication Studies, Freie Universität Berlin, 14195 Berlin, Germany

103. Tübingen School of Education, University of Tübingen, 72074 Tübingen, Germany

104. University College Dublin, Dublin 4, Ireland

105. Department of European Languages and Cultures, University of Groningen, 9712 EK Groningen, The Netherlands

106. Robert Schuman Center for Advanced Studies, European University Institute, 50133 Florence, Italy

107. Department of Security and Crime Science, University College London, London,WC1E 6BT, United Kingdom

108. Department of Migration, Leibniz Institute for Educational Trajectories, 96047 Bamberg, Germany

109. Department of Survey Design and Methodology, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany

110. Department of Sociology, University of Vienna, 1090 Vienna, Austria

111. Center for Political Science Research, KU Leuven, 3000 Leuven, Belgium

112. Interdisciplinary Social Science, Utrecht University, 3584 Utrecht, The Netherlands

113. Department of Demography, University of Vienna, 1010 Vienna, Austria

114. Institute for Social Sciences, University of Hildesheim, 31141 Hildesheim, Germany

115. Center for Social and Cultural Psychology, Université Libre de Bruxelles, 1050 Brussels, Belgium

116. Department of Psychology, University of Hagen, 58097 Hagen, Germany

117. Education and Employment, Institute for Advanced Studies, University of Vienna, Vienna, 1080 Austria

118. Research Institute for Quality of Life, Romanian Academy, 010071 Bucharest, Romania

119. Department of Sociology, Lucian Blaga University of Sibiu, 550024 Sibiu, Romania

120. Netherlands Institute for Social Research, 2500 BD The Hague, the Netherlands

121. Policy Perspectives, Citizen Perspectives, and Behaviors, Netherlands Institute for Social Research, 2594 The Hague, The Netherlands

122. Research Cluster "The Politics of Inequality", University of Konstanz, 78464 Konstanz, Germany

123. Department of Psychology, University of South Florida, Tampa, FL 33620

124. President, Leibniz Institute for the Social Sciences (GESIS), 68159 Mannheim, Germany

125. Faculty of Arts and Science, Kyushu University, Fukuoka, 819-0395, Japan

126. Department of Political Science, University of Duisburg-Essen, 47057 Duisburg, Germany

127. Institute for Employment Research, Federal Employment Agency, 90478 Nuremberg, Germany

Abstract

This study explores how researchers’ analytical choices affect the reliability of scientific findings. Most discussions of reliability problems in science focus on systematic biases. We broaden the lens to emphasize the idiosyncrasy of conscious and unconscious decisions that researchers make during data analysis. We coordinated 161 researchers in 73 research teams and observed their research decisions as they used the same data to independently test the same prominent social science hypothesis: that greater immigration reduces support for social policies among the public. In this typical case of social science research, research teams reported both widely diverging numerical findings and substantive conclusions despite identical start conditions. Researchers’ expertise, prior beliefs, and expectations barely predict the wide variation in research outcomes. More than 95% of the total variance in numerical results remains unexplained even after qualitative coding of all identifiable decisions in each team’s workflow. This reveals a universe of uncertainty that remains hidden when considering a single study in isolation. The idiosyncratic nature of how researchers’ results and conclusions varied is a previously underappreciated explanation for why many scientific hypotheses remain contested. These results call for greater epistemic humility and clarity in reporting scientific findings.

Publisher

Proceedings of the National Academy of Sciences

Subject

Multidisciplinary

Link

https://pnas.org/doi/pdf/10.1073/pnas.2203150119

Reference34 articles.

1. M. Solomon, Social Empiricism (MIT Press, 2007).

2. N. Oreskes, Why Trust Science? (Princeton University Press, 2019).

3. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015

4. Estimating the reproducibility of psychological science

5. S. Ritchie, Science Fictions: How Fraud, Bias, Negligence, and Hype Undermine the Search for Truth (Metropolitan Books, 2020).

Cited by 79 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. Behavioral risk profiling: Measuring loss aversion of individual investors;Journal of Banking & Finance;2024-11

2. The quantitative paradigm and the nature of the human mind. The replication crisis as an epistemological crisis of quantitative psychology in view of the ontic nature of the psyche;Frontiers in Psychology;2024-09-12

3. Toward a more credible assessment of the credibility of science by many-analyst studies;Proceedings of the National Academy of Sciences;2024-09-05

4. Subjective logic as a complementary tool to meta-analysis to explicitly address second-order uncertainty in research findings: A case from infant studies;Infant Behavior and Development;2024-09

5. Adversarial collaboration: An undervalued approach in behavioral science.;American Psychologist;2024-08-15