Gender and other potential biases in peer review: Analysis of 38,250 external peer review reports

Abstract

Background The Swiss National Science Foundation (SNSF) supports fundamental and use-inspired research in all disciplines. Peer reviewers assess the proposals submitted to the SNSF. We examined whether the gender of applicants and reviewers and other factors influenced the summary scores awarded. Methods We analysed 38,250 reports on 12,294 grant applications across all disciplines 2006 to 2016. Proposals were rated on a scale from 1 (=worst) to 6 (=best) by 26,836 reviewers. We used linear mixed effects regression models adjusted for research topic, applicant's age, nationality, affiliation and calendar period to examine associations, and interactions between gender of the applicant and other variables. Results In univariable analysis, male applicants received more favourable evaluation scores than female applicants (+0.19 points; 95% CI 0.14-0.23), and male reviewers awarded higher scores than female reviewers (+0.12; 95% CI 0.08-0.15). Applicant-nominated reviewers awarded higher scores than reviewers nominated by the SNSF (+0.53; 95% CI 0.50-0.56), and reviewers affiliated with research institutions outside of Switzerland more favourable scores than reviewers affiliated with Swiss institutions (+0.53; 95% CI 0.49-0.56). In multivariable analysis, differences between male and female applicants were attenuated (to +0.08; 95% CI 0.04-0.13) whereas results changed little for source of nomination and affiliation of reviewers. There was an interaction between gender of applicant and reviewer, and between gender of applicant and calendar period. Male reviewers gave higher scores than female reviewers, with a greater difference for male than for female applicants (P=0.037 from test of interaction). The gender difference increased after September 2011, when new evaluation forms were introduced (P=0.033 from test of interaction). Conclusions Our study showed that peer review of grant applications at SNSF might be prone to biases stemming from different applicant and reviewer characteristics. The SNSF abandoned the nomination of peer reviewers by applicants, and made members of panels aware of the other systematic differences in scores. The new form introduced in 2011 may inadvertently have given more emphasis to the applicant’s track record, and a revision is now under discussion. We encourage other funders to conduct similar studies, in order to improve the evidence base for rational and fair research funding.