Empirically identifying and computationally modelling the brain-behaviour relationship for human scene categorization

Abstract

AbstractHumans effortlessly make quick and accurate perceptual decisions about the nature of their immediate visual environment, such as the category of the scene they face. Previous research has revealed a rich set of cortical representations potentially underlying this feat. However, it remains unknown which of these representations are suitably formatted for decision-making. Here, we approached this question empirically and computationally, using neuroimaging and computational modelling. For the empirical part, we collected electroencephalography (EEG) data and reaction times from human participants during a scene categorization task (natural vs. man-made). We then related neural representations to behaviour using a multivariate extension of signal detection theory. We observed a correlation specifically between ∼100 ms and ∼200 ms after stimulus onset, suggesting that the neural scene representations in this time period are suitably formatted for decision-making. For the computational part, we evaluated a recurrent convolutional neural network (RCNN) as a model of brain and behaviour. Unifying our previous observations in an image-computable model, the RCNN predicted well the neural representations, the behavioural scene categorization data, as well as the relationship between them. Our results identify and computationally characterize the neural and behavioural correlates of scene categorization in humans.Significance statementCategorizing scene information is a ubiquitous and crucial task. Here we provide an empirical and computational account of scene categorization. Previous research has identified when scenes are represented in the visual processing hierarchy, but it remains unclear which of these representations are relevant for behaviour. We identified such representations between ∼100 ms and ∼200 ms after stimulus onset. We then showed that scene categorization in humans can be modelled via a recurrent convolutional neural network in a unified manner, i.e., in terms of neural and behavioural correlates, and their relationship. Together this reveals which representations underlie scene categorization behaviour and proposes a computational mechanism that implements such representations.