Decoding the physics of observed actions in the human brain

Abstract

AbstractRecognizing goal-directed actions is a computationally challenging task, requiring not only the visual analysis of body movements, but also analysis of how these movements causally impact, and thereby induce a change in, those objects targeted by an action. We tested the hypothesis that the analysis of body movements and the effects they induce relies on distinct neural representations in superior and anterior inferior parietal lobe (SPL and aIPL). In four fMRI sessions, participants observed videos of actions (e.g. breaking stick, squashing plastic bottle) along with corresponding point-light-display stick figures, pantomimes, and abstract animations of agent-object interactions (e.g. dividing or compressing a circle). Cross-decoding between actions and animations revealed that aIPL encodes abstract representations of action effect structures independent of motion and object identity. By contrast, cross-decoding between actions and point-light-displays revealed that SPL is disproportionally tuned to body movements independent of visible Interactions with objects. Lateral occipitotemporal cortex (LOTC) was sensitive to both action effects and body movements. Moreover, cross-decoding between pantomimes and animations revealed that right aIPL and LOTC represent action effects even in response to implied object interactions. These results demonstrate that parietal cortex and LOTC are tuned to physical action features, such as how body parts move in space relative to each other and how body parts interact with objects to induce a change (e.g. in position or shape/configuration). The high level of abstraction revealed by cross-decoding suggests a general neural code supporting mechanical reasoning about how entities interact with, and have effects on, each other.Significance statementUnderstanding the actions of others requires the analysis of how the body interacts with objects in the world and thereby induces a change (e.g. in location or configuration). Using fMRI in combination with multivariate pattern decoding, we show that the analysis of body movements and the effects they induce relies on distinct neural computations in superior and inferior parietal cortex. Importantly, representations of action effects were independent of movements and types of entities involved in the interaction. These findings elucidate key stages in the action recognition hierarchy and suggest that a critical role of parietal cortex in action recognition is the visuospatial analysis of how entities move and impact each other and thereby change their states.