Undermatching is a consequence of policy compression

Abstract

AbstractThe matching law describes the tendency of agents to match the ratio of choices allocated to the ratio of rewards received when choosing among multiple options (Herrnstein, 1961). Perfect matching, however, is infrequently observed. Instead, agents tend to undermatch, or bias choices towards the poorer option. Overmatching, or the tendency to bias choices towards the richer option, is rarely observed. Despite the ubiquity of undermatching, it has received an inadequate normative justification. Here, we assume agents not only seek to maximize reward, but also seek to minimize cognitive cost, which we formalize as policy complexity (the mutual information between actions and states of the environment). Policy complexity measures the extent to which an agent’s policy is state-dependent. Our theory states that capacity-constrained agents (i.e., agents that must compress their policies to reduce complexity), can only undermatch or perfectly match, but not overmatch, consistent with the empirical evidence. Moreover, we validate a novel prediction about which task conditions exaggerate undermatching. Finally, we argue that a reduction in undermatching with higher dopamine levels in patients with Parkinson’s disease is consistent with an increased policy complexity.Significance statementThe matching law describes the tendency of agents to match the ratio of choices allocated to different options to the ratio of reward received. For example, if option A yields twice as much reward as option B, matching states that agents will choose option A twice as much. However, agents typically undermatch: they choose the poorer option more frequently than expected. Here, we assume that agents seek to simultaneously maximize reward and minimize the complexity of their action policies. We show that this theory explains when and why undermatching occurs. Neurally, we show that policy complexity, and by extension undermatching, is controlled by tonic dopamine, consistent with other evidence that dopamine plays an important role in cognitive resource allocation.