Organisms use mode-switching to solve the explore-vs-exploit problem

Abstract

AbstractThe inescapable link between sensing and movement generates a conflict between producing costly movements for gathering information (“explore”) versus using previously acquired information to achieve a goal (“exploit”). Determining the optimal balance between explore and exploit is a computationally intractable problem, necessitating the use of heuristics. We looked to nature to measure and model the solutions used by organisms. Here we show that the electric fishEigenmannia virescensuses a salience-dependent mode-switching strategy to solve the explore–exploit conflict during a refuge tracking task. The fish produced distinctive non-Gaussian (i.e., non-normal) distributions of movement velocities characterized by sharp peaks for slower, task-oriented “exploit” movements and broad shoulders for faster, “explore” movements. The measures of non-normality increased in relation to increased sensory salience. Data from ten phylogenetically diverse organisms, from amoebae to humans, revealed the same distinctive distribution of movement velocities that were also modulated in relation to sensory salience. We propose a state-uncertainty based mode-switching heuristic that (1) reproduces the distinctive velocity distribution, (2) rationalizes modulation by sensory salience, and (3) outperforms the classic persistent excitation approach while using less energy. This mode-switching heuristic provides insights to purposeful exploratory behaviors in organisms as well as a framework for more efficient state estimation and control of robots.