Decoding 3D spatial location across saccades in human visual cortex

Abstract

AbstractVisual signals are initially processed as two-dimensional images on our retina, but we live in a 3D world. Depth information needs to be reconstructed from the 2D retinal images, using cues such as binocular disparity. But in daily life, we also make frequent, rapid eye movements, which alter the 2D retinal input. How do we achieve stable 3D perception across saccades? Using fMRI pattern analysis, we investigated how 3D spatial representations in human visual cortex are influenced by saccades. Participants viewed stimuli in four possible 3D locations, defined by 2D vertical position (above or below screen center) and depth position (in front of or behind central screen plane). We compared the amount of 2D and depth information in visual cortical regions during no-saccade blocks (stationary fixation) with that during saccade blocks (series of guided saccades). On no-saccade blocks, decoding of stimulus location was highly dependent on fixation position: in later visual areas we could decode both vertical and depth information across blocks that shared the same fixation position (as previously reported), but little vertical or depth information could be decoded across blocks with different fixation positions. Strikingly, the neural similarity patterns appeared tolerant to changes in fixation position during saccade blocks: despite the saccade-induced retinal and fixation changes, we could reliably decode both vertical and depth information. The findings suggest that representations of 3D spatial locations may become more tolerant of fixation positions during dynamic saccades, perhaps due to active remapping which may encourage more stable representations of the world.SignificanceThis study investigates two fundamental challenges for visual perception: how to preserve spatial information across frequent eye movements, and how to integrate binocular depth location with 2D location to form coherent 3D percepts. Aspects of these challenges have been studied in isolation, but surprisingly no studies have investigated them jointly to ask how 3D spatial representations in human visual cortex are influenced by saccades. Our fMRI pattern analysis findings highlight a potentially critical role of active, dynamic saccades on stabilizing 3D spatial representations in the brain, revealing that representations of 3D space may be modulated by eye position during sustained fixation, but could become tolerant of changes in eye position during active, dynamic saccades.