Narrative Event Segmentation in the Cortical Reservoir

Abstract

AbstractDuring continuous perception of movies or stories, awake humans display cortical activity patterns that reveal hierarchical segmentation of event structure. Sensory areas like auditory cortex display high frequency segmentation related to the stimulus, while semantic areas like posterior middle cortex display a lower frequency segmentation related to transitions between events (Baldassano et al. 2017). These hierarchical levels of segmentation are associated with different time constants for processing. Chien and Honey (2020) observed that when two groups of participants heard the same sentence in a narrative, preceded by different contexts, neural responses for the groups were initially different and then gradually aligned. The time constant for alignment followed the segmentation hierarchy: sensory cortices aligned most quickly, followed by mid-level regions, while some higher-order cortical regions took more than 10 seconds to align. These hierarchical segmentation phenomena can be considered in the context of processing related to comprehension. Uchida et al. (2021) recently described a model of discourse comprehension where word meanings are modeled by a language model pre-trained on a billion word corpus (Yamada et al 2020). During discourse comprehension, word meanings are continuously integrated in a recurrent cortical network. The model demonstrates novel discourse and inference processing, in part because of two fundamental characteristics: real-world event semantics are represented in the word embeddings, and these are integrated in a reservoir network which has an inherent gradient of functional time constants due to the recurrent connections. Here we demonstrate how this model displays hierarchical narrative event segmentation properties. The reservoir produces activation patterns that are segmented by the HMM of Baldassano et al (2017) in a manner that is comparable to that of humans. Context construction displays a continuum of time constants across reservoir neuron subset, while context forgetting has a fixed time constant across these subsets. Virtual areas formed by subgroups of reservoir neurons with faster time constants segmented with shorter events, while those with longer time constants preferred longer events. This neurocomputational recurrent neural network simulates narrative event processing as revealed by the fMRI event segmentation algorithm of Baldassano et al (2017), and provides a novel explanation of the asymmetry in narrative forgetting and construction observed by Chien and Honey (2020). The model extends the characterization of online integration processes in discourse to more extended narrative, and demonstrates how reservoir computing provides a useful model of cortical processing of narrative structure.