Abstract
AbstractThe ecology of human language is face-to-face interaction, comprising cues, like prosody, cospeech gestures, and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants whose electroencephalographic activity was recorded. We quantified each cue and determined their effect on a well-established electroencephalographic marker of cognitive load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, brain responses were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other available cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.
Publisher
Cold Spring Harbor Laboratory
Cited by
5 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献