Manual gestures facilitate learning lexical stress by modulating auditory neural responses

Abstract

AbstractGestures accompany speech and facilitate communication and learning. Most previous studies have demonstrated the effects of gestures on learning semantics, yet how gestures facilitate learning low- and intermediate-level speech features is unclear. The present study investigated the effects of manual gestures on learning lexical stress, a phonological-lexical feature that is foreign to native Mandarin speakers. Across a series of experiments, we demonstrated that the gestures with representational relations to auditory stimuli in terms of covaried amplitude modulation facilitated the learning of lexical stress for both familiar (English) and unknown (Russian) languages, but not for pseudowords that lack phonotactic properties. Interestingly, gestures with amplitude trajectory matching the stress benefited the learning of trained words, whereas gestures that only matched the timing of syllable segments but not amplitude variation generalized the learning effects to untrained stimuli. Furthermore, in the EEG experiment, we found that gesture-accompanied learning was associated with power increase and inter-trial phase coherence (ITC) decrease in the theta band at the time windows corresponding to the stress positions. These results suggest that the facilitatory effects of gestures on lexical stress learning depend on the specificity of cross-modal feature mapping at the phonological level, mediated by the neural modulation in early perceptual responses.