Community review: a robust and scalable selection system for resource allocation within open science and innovation communities

Author:

Graham Chris L.B.ORCID,Landrain Thomas E.,Vjestica Amber,Masselot Camille,Lawton Elliot,Blondel Leo,Haenal Luca,Greshake Tzovaras Bastian,Santolini MarcORCID

Abstract

Resource allocation is essential to the selection and implementation of innovative projects in science and technology. With large stakes involved in concentrating large fundings over a few promising projects, current “winner-take-all” models for grant applications are time-intensive endeavours that mobilise significant researcher time in writing extensive project proposals, and rely on the availability of a few time-saturated volunteer experts. Such processes usually carry over several months, resulting in high effective costs compared to expected benefits. Faced with the need for a rapid response to the COVID-19 pandemic in 2020, we devised an agile “community review” system, similar to distributed peer review (DPR) systems, to allocate micro-grants for the fast prototyping of innovative solutions. Here we describe and evaluate the implementation of this community review across 147 projects from the “Just One Giant Lab’s OpenCOVID19 initiative” and “Helpful Engineering” open research communities. The community review process uses granular review forms and requires the participation of grant applicants in the review process. We show that this system is fast, with a median duration of 10 days, scalable, with a median of 4 reviewers per project independent of the total number of projects, and fair, with project rankings highly preserved after the synthetic removal of reviewers. We investigate potential bias introduced by involving applicants in the process, and find that review scores from both applicants and non-applicants have a similar correlation of r=0.28 with other reviews within a project, matching previous observations using traditional approaches. Finally, we find that the ability of projects to apply to several rounds allows to both foster the further implementation of successful early prototypes, as well as provide a pathway to constructively improve an initially failing proposal in an agile manner.  This study quantitatively highlights the benefits of a frugal community review system for agile resource allocation.

Funder

Agence Nationale de la Recherche

AXA Research Fund

Publisher

F1000 Research Ltd

Subject

General Pharmacology, Toxicology and Pharmaceutics,General Immunology and Microbiology,General Biochemistry, Genetics and Molecular Biology,General Medicine

Reference28 articles.

1. The Decision-Making Constraints and Processes of Grant Peer Review, and Their Effects on the Review Outcome.;L Langfeldt;Soc. Stud. Sci.,2001 Dec 1

2. Australia’s grant system wastes time.;D Herbert;Nature.,2013 Mar

3. Cost of the NSERC Science Grant Peer Review System Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant.;R Gordon;Account. Res.,2009 Feb 27

4. Peer Review or Lottery? A Critical Analysis of Two Different Forms of Decision-making Mechanisms for Allocation of Research Grants.;L Roumbanis;Sci. Technol. Hum. Values.,2019 Nov 1

5. Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports.;A Severin;PeerJ. Inc.,2019 Jun cited 2021 Jul 9

同舟云学术

1.学者识别学者识别

2.学术分析学术分析

3.人才评估人才评估

"同舟云学术"是以全球学者为主线,采集、加工和组织学术论文而形成的新型学术文献查询和分析系统,可以对全球学者进行文献检索和人才价值评估。用户可以通过关注某些学科领域的顶尖人物而持续追踪该领域的学科进展和研究前沿。经过近期的数据扩容,当前同舟云学术共收录了国内外主流学术期刊6万余种,收集的期刊论文及会议论文总量共计约1.5亿篇,并以每天添加12000余篇中外论文的速度递增。我们也可以为用户提供个性化、定制化的学者数据。欢迎来电咨询!咨询电话:010-8811{复制后删除}0370

www.globalauthorid.com

TOP

Copyright © 2019-2024 北京同舟云网络信息技术有限公司
京公网安备11010802033243号  京ICP备18003416号-3