Anterior cingulate cortex represents action-state predictions and causally mediates model-based reinforcement learning in a two-step decision task

Abstract

SummaryThe anterior cingulate cortex (ACC) is implicated in learning the value of actions, but it remains poorly understood whether and how it contributes to model-based mechanisms that use action-state predictions and afford behavioural flexibility. To isolate these mechanisms, we developed a multi-step decision task for mice in which both action-state transition probabilities and reward probabilities changed over time. Calcium imaging revealed ramps of choice-selective neuronal activity, followed by an evolving representation of the state reached and trial outcome, with different neuronal populations representing reward in different states. ACC neurons represented the current action-state transition structure, whether state transitions were expected or surprising, and the predicted state given chosen action. Optogenetic inhibition of ACC blocked the influence of action-state transitions on subsequent choice, without affecting the influence of rewards. These data support a role for ACC in model-based reinforcement learning, specifically in using action-state transitions to guide subsequent choice.HighlightsA novel two-step task disambiguates model-based and model-free RL in mice.ACC represents all trial events, reward representation is contextualised by state.ACC represents action-state transition structure, predicted states, and surprise.Inhibiting ACC impedes action-state transitions from influencing subsequent choice.