Rewarding Numbers: Quantification Premia and Evaluative Convergence

Abstract

The author advances the theory that when evaluation bears high stakes and is subject to high uncertainty of quality, a greater presence of numbers in the evaluated materials positively influences the evaluator’s assessment of the quality of the evaluated object and leads to less variance in the overall assessment of quality by evaluators. The author explores these ideas in a case study of the MacArthur Foundation’s 2016–2017 $100 million winner-take-all grant competition for nonprofit organizations and tests them using judges’ numeric scores and comments together with information from the application materials, tax records, and previous funding histories of the applicant organizations. In this competition, organizations that included more numbers of any kind in their application materials received on average higher scores. Furthermore, the independent judges on the nondeliberative panel were more likely to give the applicants similar scores. Quantification thus both carries a premium—it predicts higher scores—and produces evaluative convergence.