Cross-Modal Effects in Speech Perception

Abstract

Speech research during recent years has moved progressively away from its traditional focus on audition toward a more multisensory approach. In addition to audition and vision, many somatosenses including proprioception, pressure, vibration, and aerotactile sensation are all highly relevant modalities for experiencing and/or conveying speech. In this article, we review both long-standing cross-modal effects stemming from decades of audiovisual speech research and new findings related to somatosensory effects. Cross-modal effects in speech perception to date have been found to be constrained by temporal congruence and signal relevance, but appear to be unconstrained by spatial congruence. The literature reveals that, far from taking place in a one-, two-, or even three-dimensional space, speech occupies a highly multidimensional sensory space. We argue that future research in cross-modal effects should expand to consider each of these modalities both separately and in combination with other modalities in speech.