Dorsal anterior cingulate-midbrain ensemble as a reinforcement meta-learner

Abstract

AbstractThe dorsal anterior cingulate cortex (dACC) is central in higher-order cognition and behavioural flexibility. The computational nature of this region, however, has remained elusive. Here we propose a new model – the Reinforcement Meta Learner (RML) – based on the bidirectional anatomical connections of the ACC with midbrain catecholamine nuclei (VTA and LC). In this circuit, dACC learns which actions are valuable and acts accordingly. Crucially, this mechanism is optimized by recurrent connectivity with the midbrain: Midbrain catecholamines provide modulatory signals to dACC, controlling its internal parameters (e.g. learning rate), while these parameter modulations are in turn optimized by dACC afferents to the midbrain. This closed-loop system generates emergent (i.e., homunculus-free) control and supports learning to solve hierarchical decision problems without having an intrinsic hierarchical structure itself. Further, it can be combined with other cortical modules to optimize the processing of these modules. We outline how the RML solves the current theoretical stalemate on dACC by assimilating various previous proposals on ACC functioning, and how it captures critical empirical findings from an unprecedented range of domains (stability/plasticity balance, effort processing, working memory, and higher-order classical and instrumental conditioning).