REFORMS: Consensus-based Recommendations for Machine-learning-based Science-Reference-Cited by-同舟云学术

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Published:2024-05-03 Issue:18 Volume:10 Page:
ISSN:2375-2548
Container-title:Science Advances
language:en
Short-container-title:Sci. Adv.

Author:

Kapoor Sayash¹²^ORCID,Cantrell Emily M.³⁴^ORCID,Peng Kenny⁵^ORCID,Pham Thanh Hien¹²,Bail Christopher A.⁶⁷⁸^ORCID,Gundersen Odd Erik⁹¹⁰^ORCID,Hofman Jake M.¹¹^ORCID,Hullman Jessica¹²^ORCID,Lones Michael A.¹³^ORCID,Malik Momin M.¹⁴¹⁵¹⁶^ORCID,Nanayakkara Priyanka¹²¹⁷,Poldrack Russell A.¹⁸^ORCID,Raji Inioluwa Deborah¹⁹,Roberts Michael²⁰²¹^ORCID,Salganik Matthew J.²³²²^ORCID,Serra-Garcia Marta²³^ORCID,Stewart Brandon M.²³²²²⁴^ORCID,Vandewiele Gilles²⁵^ORCID,Narayanan Arvind¹²^ORCID

Affiliation:

1. Department of Computer Science, Princeton University, Princeton, NJ 08544, USA.

2. Center for Information Technology Policy, Princeton University, Princeton, NJ 08544, USA.

3. Department of Sociology, Princeton University, Princeton, NJ 08544, USA.

4. School of Public and International Affairs, Princeton University, Princeton, NJ 08544, USA.

5. Department of Computer Science, Cornell University, Ithaca, NY 14850, USA.

6. Department of Sociology, Duke University, Durham, NC 27708, USA.

7. Department of Political Science, Duke University, Durham, NC 27708, USA.

8. Sanford School of Public Policy, Duke University, Durham, NC 27708, USA.

9. Department of Computer Science, Norwegian University of Science and Technology, Trondheim, Norway.

10. Aneo AS, Trondheim, Norway.

11. Microsoft Research, New York, NY 10012, USA.

12. Department of Computer Science, Northwestern University, Evanston, IL 60208, USA.

13. School of Mathematical and Computer Sciences, Heriot-Watt University, Edinburgh, UK.

14. Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA.

15. School of Social Policy & Practice, University of Pennsylvania, Philadelphia, PA 19104, USA.

16. Institute in Critical Quantitative, Computational, & Mixed Methodologies, Johns Hopkins University, Baltimore, MD 21218, USA.

17. Department of Communication Studies, Northwestern University, Evanston, IL 60208, USA.

18. Department of Psychology, Stanford University, Stanford, CA 94305, USA.

19. Department of Computer Science, University of California, Berkeley, Berkeley, CA 94720, USA.

20. Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK.

21. Department of Medicine, University of Cambridge, Cambridge, UK.

22. Office of Population Research, Princeton University, Princeton, NJ 08544, USA.

23. Rady School of Management, University of California, San Diego, La Jolla, CA 92093, USA.

24. Department of Politics, Princeton University, Princeton, NJ 08544, USA.

25. Department of Information Technology, Ghent University, Ghent, Belgium.

Abstract

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

Publisher

American Association for the Advancement of Science (AAAS)

Link

https://www.science.org/doi/pdf/10.1126/sciadv.adk3452

Reference202 articles.

1. Machine Learning Methods That Economists Should Know About

2. Supervised Machine Learning for Population Genetics: A New Paradigm

3. Applications of machine learning in animal behaviour studies

4. Machine learning, statistical learning and the future of biological research in psychiatry

5. Big Data Methods

Cited by 4 articles. 订阅此论文施引文献订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献

1. LLMscreen: A Python Package for Systematic Review Screening of Scientific Texts Using Prompt Engineering;2024-09-11

2. The reanimation of pseudoscience in machine learning and its ethical repercussions;Patterns;2024-09

3. A Workflow for Building Computationally Rational Models of Human Behavior;Computational Brain & Behavior;2024-08-15

4. Avoiding common machine learning pitfalls;Patterns;2024-08