Perceptual uncertainty explains activation differences between audiovisual congruent speech and McGurk stimuli

Abstract

AbstractFace-to-face communication relies on the integration of acoustic speech signals with the corresponding facial articulations. While the McGurk illusion is widely used as an index of audiovisual speech integration, critics argue that it arises from perceptual processes that differ categorically from natural speech recognition. Conversely, Bayesian theoretical frameworks suggest that both the illusory McGurk and the veridical audiovisual congruent speech percepts result from probabilistic inference based on noisy sensory signals. According to these models, the inter-sensory conflict in McGurk stimuli may only increase observers’ perceptual uncertainty. This functional magnetic resonance imaging (fMRI) study presented participants (20 male and 24 female) with audiovisual congruent, incongruent, and McGurk stimuli along with their unisensory counterparts in a syllable categorization task. Behaviorally, observers’ response entropy was greater for McGurk compared to congruent audiovisual stimuli. At the neural level, McGurk stimuli increased activations in a widespread neural system, extending from the inferior frontal sulci (IFS) to the pre-supplementary motor area (pre-SMA) and insulae, typically involved in cognitive control processes. Crucially, in line with Bayesian theories these activation increases were fully accounted for by observers’ perceptual uncertainty as measured by their response entropy. Our findings suggest that McGurk and congruent speech processing rely on shared neural mechanisms, thereby supporting the McGurk illusion as a valid measure of natural audiovisual speech perception.Significance StatementEffective face-to-face communication relies on integrating acoustic speech signals with the corresponding facial articulations. While McGurk illusion is extensively used to study audiovisual speech perception, recent critiques argue that it may be categorically different from typical speech recognition because of the conflict between the audiovisual inputs. This study demonstrates that McGurk stimuli increase activations in a network of regions typically involved in cognitive control. Crucially, the activation differences between McGurk and normal speech stimuli could be fully accounted for by the variation in observers’ perceptual uncertainties. Our results suggest that McGurk and congruent audiovisual speech stimuli rely on shared neural mechanisms – thereby supporting the validity of the McGurk illusion as a tool for studying natural audiovisual speech perception.