Lip movements and lexical features improve speech tracking differently for clear and multi-speaker speech

Abstract

AbstractVisual speech plays a powerful role in facilitating auditory speech processing and has been a publicly noticed topic with the wide usage of face masks during the Covid-19 pandemic. In a previous magnetoencephalography (MEG) study we showed that occluding the mouth area significantly impairs neural speech tracking. To rule out the possibility that this deterioration is due to degraded sound quality, in the present follow-up study, we presented participants with audiovisual (AV) and audio-only (A) speech. We further independently manipulated the trials by adding a face mask and a distractor speaker. Our results clearly show that face masks only affect speech tracking in AV conditions, not in A conditions. This shows that face masks indeed primarily impact speech processing by blocking visual speech and not by acoustic degradation. Furthermore, we observe differences in the speech features that are used for visual speech processing. On the one hand, processing in clear speech, but not in noisy speech, is profiting more from lexical unit features (phonemes and word onsets) hinting at improved phoneme discrimination. On the other hand, we observe an improvement in speech tracking driven by the modulations of the lip area in clear speech and conditions with a distractor speaker, which might aid by providing temporal cues for subsequent auditory processing. With this work, we highlight the effects of face masks in AV speech tracking and show two separate ways how visual speech might support successful speech processing.