Does the inclusion of non-academic reviewers make any difference for grant impact panels?

Abstract

Abstract Broader impact of scientific research beyond academia has become increasingly important in research evaluation. To evaluate broader impact of research proposals, some funding agencies compose mixed panels that include peer experts and non-academic stakeholders. Whether and how non-academic reviewers bring any difference to panel discussions has been understudied. We analysed 164 review reports (2014–6) from the Investigators Programme (funding Science, Technology, Engineering, and Mathematics fields) at Science Foundation Ireland, where two types of panels, with and without non-academics, were composed for impact assessments. We find that the mixed panel reviews were longer and touched upon broader and more concrete impact topics. Also, mixed panels commented on causality and attribution of impact towards characteristics of applicants and research process more than scientific excellence. A survey of the same reviewer pool supplements our understanding of the pros and cons of the inclusion of non-academic reviewers. We discuss some policy recommendations for funding agencies to organise review panels.