Journal Reviewer Ratings: Issues of Particularistic Bias, Agreement, and Predictive Validity Within the Manuscript Review Process

Abstract

Reviewer evaluations and recommendations for 853 manuscript submissions, over a span of 4 years, are analyzed for evidence of particularistic bias, reviewer agreement, and predictive validity for forecasting a published manuscript's citation impact. Attributes of the submitters, their affiliated institutions, and the reviewers have little consistent association with reviewers' recommendations or editorial decision outcomes. Furthermore, reviewers' recommendations demonstrate a reasonable degree of agreement. However, neither reviewers' evaluative ratings across five dimensions nor publication recommendations can predict the number of citations that a published article subsequently receives. Strengths and limitations of various features of the manuscript review process, as well as the importance of monitoring the process for particularistic biases and evidence of predictive validity, are discussed.