Investigation of contributions from cortical and subcortical brain structures for speech decoding

Abstract

AbstractLanguage impairments often arise from severe neurological disorders, prompting the development of neural prosthetics based on electrophysiological signals for the restoration of comprehensible language information. Previous decoding efforts have focused mainly on signals from the cerebral cortex, neglecting the potential contributions of subcortical brain structures to speech decoding in brain-computer interfaces (BCIs). This study aims to explore the role of subcortical structures for speech decoding by utilizing stereotactic electroencephalography (sEEG). Two native Mandarin Chinese speakers, who underwent sEEG implantation for pharmaco-resistant epilepsy, participated in this study. sEEG contacts were primarily located in the superior temporal gyrus, middle temporal gyrus, inferior temporal gyrus, thalamus, hippocampus, insular gyrus, amygdala, and parahippocampal gyrus. The participants were asked to read Chinese text, which included 407 Chinese characters (covering all Chinese syllables), displayed on a screen after receiving prompts. 1-30, 30-70 and 70-150 Hz frequency band powers of sEEG signals were used as key features. A deep learning model based on long short-term memory (LSTM) was developed to evaluate the contribution of different brain structures during encoding of speech. Prediction of speech characteristics of consonants (articulatory place and manner) and tone within single words based on the selected features and electrode contact locations was made. Cortical signals were generally better at articulatory place prediction (86.5% accuracy, chance level = 12.5%), while cortical and subcortical signals predicted articulatory manner at similar level (51.5% vs 51.7% accuracy, respectively, chance level = 14.3%). Subcortical signals generated better prediction for tone (around 58.3% accuracy, chance level = 25%). Superior temporal gyrus remains highly relevant during speech decoding for both consonants and tone. Prediction reached the highest level when cortical and subcortical inputs were combined, especially for tone prediction. Our findings indicate that both cortical and subcortical structures can play crucial roles for speech decoding, each contributing to different aspects of speech.