Abstract
AbstractTo transform speech into words, the human brain must accommodate variability across utterances in intonation, speech rate, volume, accents and so on. A promising approach to explaining this process has been to model electroencephalogram (EEG) recordings of brain responses to speech. Contemporary models typically invoke speech categories (e.g. phonemes) as an intermediary representational stage between sounds and words. However, such categorical models are typically hand-crafted and therefore incomplete because they cannot speak to the neural computations that putatively underpin categorization. By providing end-to-end accounts of speech-to-language transformation, new deep-learning systems could enable more complete brain models. We here model EEG recordings of audiobook comprehension with the deep-learning system Whisper. We find that (1) Whisper provides an accurate, self-contained EEG model of speech-to-language transformation; (2) EEG modeling is more accurate when including prior speech context, which pure categorical models do not support; (3) EEG signatures of speech-to-language transformation depend on listener-attention.
Publisher
Cold Spring Harbor Laboratory
Reference64 articles.
1. Speech comprehension is correlated with temporal response patterns recorded from auditory cortex
2. Human Cortical Responses to the Speech Envelope
3. Deep Artificial Neural Networks Reveal a Distributed Cortical Network Encoding Propositional Sentence-Level Meaning
4. Antonello, R. , Vaidya, A. , & Huth, A. G. 2023. Scaling laws for language encoding models in fMRI. arXiv preprint arXiv:2305.11863
5. Baevski A , Zhou H , Mohamed A , Auli M. 2020. wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems 2020 Dec 6(pp. 12449–12460).