Subjective evidence evaluation survey for many-analysts studies

Author:

Sarafoglou Alexandra1ORCID,Hoogeveen Suzanne2ORCID,van den Bergh Don1,Aczel Balazs3ORCID,Albers Casper J.4,Althoff Tim5,Botvinik-Nezer Rotem67,Busch Niko A.8,Cataldo Andrea M.910,Devezer Berna11ORCID,van Dongen Noah N. N.1,Dreber Anna1213ORCID,Fried Eiko I.14,Hoekstra Rink15ORCID,Hoffman Sabine16ORCID,Holzmeister Felix13,Huber Jürgen13,Huntington-Klein Nick17,Ioannidis John18ORCID,Johannesson Magnus12,Kirchler Michael13,Loken Eric19,Mangin Jan-Francois2021,Matzke Dora1,Menkveld Albert J.22,Nilsonne Gustav23ORCID,van Ravenzwaaij Don4ORCID,Schweinsberg Martin24,Schulz-Kuempel Hannah2526,Shanks David R.27ORCID,Simons Daniel J.28,Spellman Barbara A.29,Stoevenbelt Andrea H.15,Szaszi Barnabas3,Trübutschek Darinka30,Tuerlinckx Francis31,Uhlmann Eric L.32,Vanpaemel Wolf31ORCID,Wicherts Jelte33,Wagenmakers Eric-Jan1

Affiliation:

1. Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands

2. Utrecht University, Utrecht, The Netherlands

3. Institute of Psychology, ELTE Eötvös Lorénd University, Budapest, Hungary

4. Heymans Institute for Psychological Research, University of Groningen, Groningen, The Netherlands

5. Allen School of Computer Science and Engineering, University of Washington, Seattle, WA, USA

6. Hebrew University of Jerusalem, Jerusalem, Israel

7. Dartmouth College, Hanover, NH, USA

8. Institute for Psychology, University of Münster, Münster, Germany

9. Center for Depression, Anxiety and Stress Research, McLean Hospital, Belmont, MA, USA

10. Department of Psychiatry, Harvard Medical School, Boston, MA, USA

11. Department of Business, University of Idaho, Moscow, ID, USA

12. Stockholm School of Economics, Stockholm, Sweden

13. University of Innsbruck, Innsbruck, Tirol, Austria

14. Department of Psychology, Leiden University, Leiden, The Netherlands

15. Nieuwenhuis Institute for Educational Research, University of Groningen, Groningen, The Netherlands

16. Department of Statistics, Ludwig-Maximilians-Universität München, Munchen, Bayern, Germany

17. Seattle University, Seattle, WA, USA

18. Meta-Research Innovation Center at Stanford (METRICS) and Departments of Medicine, of Epidemiology and of Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, CA, USA

19. University of Conneticut, Storrs, CT, USA

20. University Paris-Saclay, Gif-sur-Yvette, France

21. Neurospin CEA, Gif-sur-Yvette, Île-de-France, France

22. Vrije Universiteit Amsterdam, Amsterdam, Noord-Holland, The Netherlands

23. Karolinska Institutet, Solna, Sweden

24. ESMT Berlin, Berlin, Germany

25. Department of Statistics and The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany

26. The Institute for Medical Information Processing, Biometry, and Epidemiology, LMU Munich, Munchen, Bayern, Germany

27. Division of Psychology and Language Sciences, University College London, 26 Bedford Way, London WC1H 0AP, UK

28. University of Illinois—Urbana-Champaign, Urbana, IL, USA

29. School of Law, University of Virginia, 580 Massie Road, Charlottesville, VA, USA

30. Max Planck Institute for Empirical Aesthetics, Frankfurt am Main, Germany

31. University of Leuven, Leuven, Belgium

32. INSEAD, Fontainebleau, Île-de-France, France

33. Department of Methodology and Statistics, Tilburg University, Tilburg, The Netherlands

Abstract

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

Funder

National Science Foundation

HORIZON EUROPE Marie Sklodowska-Curie Actions

Nederlandse Organisatie voor Wetenschappelijk Onderzoek

Deutsche Forschungsgemeinschaft

Amsterdam Brain and Cognition

National Institute of General Medical Sciences

UK Economic and Social Research Council

Knut and Alice Wallenberg Foundation

Institut Européen d'Administration des Affaires

Jan Wallanders och Tom Hedelius Stiftelse samt Tore Browaldhs Stiftelse

Marcus och Amalia Wallenbergs minnesfond

Publisher

The Royal Society

Cited by 1 articles. 订阅此论文施引文献 订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献

1. Heterogeneity in effect size estimates;Proceedings of the National Academy of Sciences;2024-07-30

同舟云学术

1.学者识别学者识别

2.学术分析学术分析

3.人才评估人才评估

"同舟云学术"是以全球学者为主线,采集、加工和组织学术论文而形成的新型学术文献查询和分析系统,可以对全球学者进行文献检索和人才价值评估。用户可以通过关注某些学科领域的顶尖人物而持续追踪该领域的学科进展和研究前沿。经过近期的数据扩容,当前同舟云学术共收录了国内外主流学术期刊6万余种,收集的期刊论文及会议论文总量共计约1.5亿篇,并以每天添加12000余篇中外论文的速度递增。我们也可以为用户提供个性化、定制化的学者数据。欢迎来电咨询!咨询电话:010-8811{复制后删除}0370

www.globalauthorid.com

TOP

Copyright © 2019-2024 北京同舟云网络信息技术有限公司
京公网安备11010802033243号  京ICP备18003416号-3