Observing Many Researchers Using the Same Data and Hypothesis Reveals a Hidden Universe of Uncertainty
Author:
Breznau NateORCID, Rinke Eike MarkORCID, Wuttke AlexanderORCID, Adem Muna, Adriaans Jule, Alvarez-Benjumea Amalia, Andersen Henrik KennethORCID, Auer DanielORCID, Azevedo FlavioORCID, Bahnsen Oke, Balzer Dave, Bauer GerritORCID, Bauer Paul, Baumann Markus, Baute Sharon, Benoit Verena, Bernauer Julian, Berning Carl, Berthold Anna, Bethke Felix S., Biegert Thomas, Blinzler Katharina, Blumenberg Johannes, Bobzien Licia, Bohman Andrea, Bol Thijs, Bostic Amie, Brzozowska ZuzannaORCID, Burgdorf Katharina, Burger Kaspar, Busch Kathrin, Castillo Juan CarlosORCID, Chan Nathan, Christmann Pablo, Connelly Roxanne, Czymara Christian S.ORCID, Damian Elena, Ecker Alejandro, Edelmann AchimORCID, Eger Maureen A.ORCID, Ellerbrock Simon, Forke Anna, Forster Andrea GabrieleORCID, Gaasendam Chris, Gavras KonstantinORCID, Gayle Vernon, Gessler Theresa, Gnambs TimoORCID, Godefroidt Amélie, Grömping Max, Groß Martin, Gruber Stefan, Gummer Tobias, Hadjar Andreas, Heisig Jan Paul, Hellmeier SebastianORCID, Heyne Stefanie, Hirsch MagdalenaORCID, Hjerm Mikael, Hochman Oshrat, Hövermann Andreas, Hunger Sophia, Hunkler Christian, Huth Nora, Ignacz Zsofia, Jacobs Laura, Jacobsen Jannes, Jaeger Bastian, Jungkunz SebastianORCID, Jungmann NilsORCID, Kauff Mathias, Kleinert Manuel, Klinger Julia, Kolb Jan-Philipp, Kołczyńska Marta, Kuk John Seungmin, Kunißen Katharina, Sinatra Dafina Kurti, Greinert Alexander, Lersch Philipp M., Löbel Lea-Maria, Lutscher Philipp, Mader Matthias, Madia Joan ElielORCID, Malancu Natalia, Maldonado Luis, Marahrens Helge, Martin Nicole, Martinez Paul, Mayerl Jochen, MAYORGA OSCAR JoseORCID, McManus PatriciaORCID, Wagner KyleORCID, Meeusen Cecil, Meierrieks Daniel, Mellon JonathanORCID, Merhout FriedolinORCID, Merk Samuel, Meyer DanielORCID, Micheli Leticia, Mijs Jonathan J.B.ORCID, Moya Cristóbal, Neunhoeffer Marcel, Nüst DanielORCID, Nygård OlavORCID, Ochsenfeld Fabian, Otte Gunnar, Pechenkina AnnaORCID, Prosser Christopher, Raes Louis, Ralston Kevin, Ramos Miguel, Roets ArneORCID, Rogers Jonathan, Ropers Guido, Samuel RobinORCID, Sand Gergor, Schachter Ariela, Schaeffer MerlinORCID, Schieferdecker David, Schlueter Elmar, Schmidt KatjaORCID, Schmidt Regine, Schmidt-Catran Alexander, Schmiedeberg Claudia, Schneider JürgenORCID, Schoonvelde Martijn, Schulte-Cloos JuliaORCID, Schumann Sandy, Schunck Reinhard, Schupp Juergen, Seuring Julian, Silber Henning, Sleegers Willem W. A., Sonntag Nico, Staudt Alexander, Steiber NadiaORCID, Steiner NilsORCID, Sternberg Sebastian, Stiers Dieter, Stojmenovska Dragana, Storz Nora, Striessnig ErichORCID, Stroppe Anne-Kathrin, Teltemann Janna, Tibajev Andrey, Tung Brian B., Vagni Giacomo, Van Assche JasperORCID, van der Linden Meta, van der Noll JolandaORCID, Van Hootegem Arno, Vogtenhuber StefanORCID, Voicu Bogdan, Wagemans Fieke, Wehl NadjaORCID, Werner Hannah, Wiernik Brenton M.ORCID, Winter FabianORCID, Wolf ChristofORCID, Yamada Yuki, Zhang Nan, Ziller Conrad, Zins Stefan, Żółtak Tomasz, Nguyen Hung Hoang VietORCID
Abstract
How does noise generated by researcher decisions undermine the credibility of science? We test this by observing all decisions made among 73 research teams as they independently conduct studies on the same hypothesis with identical starting data. We find excessive variation of outcomes. When combined, the 107 observed research decisions taken across teams explained at most 2.6% of the total variance in effect sizes and 10% of the deviance in subjective conclusions. Expertise, prior beliefs and attitudes of the researchers explain even less. Each model deployed to test the hypothesis was unique, which highlights a vast universe of research design variability that is normally hidden from view and suggests humility when presenting and interpreting scientific findings.
Publisher
Center for Open Science
Cited by
15 articles.
订阅此论文施引文献
订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献
|
|