Affect, Anticipation, and Adaptation: Affect-Controlled Selection of Anticipatory Simulation in Artificial Adaptive Agents

Abstract

Emotion plays an important role in thinking. In this article we study affective control of the amount of simulated anticipatory behavior in adaptive agents using a computational model. Our approach is based on model-based reinforcement learning (RL) and inspired by the simulation hypothesis (Cotterill, 2001; Hesslow, 2002). The simulation hypothesis states that thinking is internal simulation of behavior using the same sensory-motor systems as those used for overt behavior. Here, we study the adaptiveness of an artificial agent, when action-selection bias is induced by an affect-controlled amount of simulated anticipatory behavior . To this end, we introduce an affect-controlled simulation-selection mechanism that uses the predictions of the agent's RL model to select anticipatory behaviors for simulation. Based on eXperiments with adaptive agents in two nondeterministic partially observable grid-worlds we conclude that (1) internal simulation has an adaptive benefit and (2) affective control can reduce the amount of simulation needed for this benefit. This is specifically the case if the following relation holds: positive affect decreases the amount of simulation towards simulating the best potential neXt action, while negative affect increases the amount of simulation towards simulating all potential neXt actions. In essence we use artificial affect to control mental eXploration versus eXploitations. Thus, agents “feeling positive” can think ahead in a narrow sense and free up working memory resources, while agents “feeling negative” must think ahead in a broad sense and maXimize usage of working memory. Our results are consistent with several psychological findings on the relation between affect and learning, and contribute to answering the question of when positive versus negative affect is useful during adaptation.