When Less Is More: How Statistical Discrimination Can Decrease Predictive Accuracy

Abstract

Discrimination is a pervasive aspect of modern society and human relations. Statistical discrimination theory suggests that profit-maximizing employers should use all the information about job candidates, including information about group membership (e.g., race or gender), to make accurate predictions. In contrast, research on heuristics in psychology suggests that using less information can be better. Drawing on research on heuristics, we show that even small amounts of inconsistency can make predictions using group membership less accurate than predictions that do not use this information. That is, whereas statistical discrimination theory implies that better predictions can be achieved by using all available information about an individual (including group characteristics that may be correlated with but do not cause performance), our model shows that using all available information only improves predictive accuracy under a very specific set of conditions, thus suggesting that statistical discrimination often results in worse predictions. By understanding when statistical discrimination improves or worsens predictions, our work cautions decision makers and uncovers paths toward reducing the occurrence of situations in which statistical discrimination benefits predictive accuracy, thus reducing its pervasiveness in society.