Cortical processing of reference in language revealed by computational models

Abstract

AbstractHuman language processing involves not only combining word meanings in accordance with semantic and syntactic constraints, but also figuring out who and what is being referred to. Here we present a first study towards a mechanistic understanding of the neural basis for referential processing. Using both functional MRI and magnetoencephalography (MEG), we identified a consistent increase of activity in a network spanning the anterior and posterior left middle temporal gyrus and the angular gyrus for pronoun processing during naturalistic listening for both English and Chinese speakers. We then adopted a “reverse-engineering” approach to examine the cognitive processes underlying pronoun resolution. We evaluated the neural fit of three symbolic models that each formalizes a different strand of explanation for pronoun resolution in the cognitive and linguistic literature, as well as two deep neural network models with an LSTM or a Transformer architecture. Our results favor the memory-based symbolic model, suggesting a domain-general mechanism of pronoun resolution that resembles memory retrieval.