Dynamic refinement of behavioral structure mediates dopamine-dependent credit assignment

Abstract

AbstractAnimals exhibit a diverse behavioral repertoire when exploring new environments and can learn which actions or action sequences produce positive outcomes. Dopamine release upon encountering reward is critical for reinforcing reward-producing actions1–3. However, it has been challenging to understand how credit is assigned to the exact action that produced dopamine release during continuous behavior. We investigated this problem with a novel self-stimulation paradigm in which specific spontaneous movements triggered optogenetic stimulation of dopaminergic neurons. We uncovered that dopamine self-stimulation rapidly and dynamically changes the structure of the entire behavioral repertoire. Initial stimulations reinforced not only the stimulation-producing target action, but also actions similar to the target and actions that occurred a few seconds before stimulation. Repeated pairings led to gradual refinement of the behavioral repertoire leading animals to home in on the target action. Reinforcement of action sequences revealed further temporal dependencies of behavioral refinement. Action pairs that tend to be spontaneously separated by long time intervals promoted a stepwise credit assignment, with early refinement of actions most proximal to stimulation and subsequent refinement of more distal actions. Thus, a retrospective reinforcement mechanism promotes gradual refinement of the entire behavioral repertoire to assign credit to specific actions and action sequences that lead to dopamine release.