Methods for computing the maximum performance of computational models of fMRI responses

Abstract

AbstractComputational neuroimaging methods aim to predict brain responses (measured e.g. with functional magnetic resonance imaging [fMRI]) on the basis of stimulus features obtained through computational models. The accuracy of such prediction is used as an indicator of how well the model describes the computations underlying the brain function that is being considered. However, the prediction accuracy is bounded by the proportion of the variance of the brain response which is related to the measurement noise and not with the stimuli (or cognitive functions). The bound to the performance of a computational model to the prediction of brain responses has been referred to as the noise ceiling. In previous neuroimaging applications two methods have been proposed for estimating the noise ceiling based on either a split-half procedure or Monte Carlo simulations. These methods make different assumptions over the nature of the effects underlying the data, and, importantly, their relation has not been clarified yet. Here, we use a two-level generative framework to formally describe the partition between the variance of measurement noise and the stimulus related variance. In this framework we derive an analytical form for the noise ceiling that does not require computationally expensive simulations or a splitting procedure that reduce the amount of data. We describe the relation between the newly introduced noise ceiling estimator and the previous methods for variable levels of measurements noise using simulated data. Additionally, as the relation to the noise ceiling is used to make conclusions on the validity of a model with respect to others, we evaluate the effect the interplay between regularization (often used to estimate model fits to the data when the number of computational features in the model is large) and model complexity on the performance with respect to the noise ceiling. Finally, we show the differences between the methods on real fMRI data acquired at 7 Tesla. We demonstrate that while the split half estimator provides a pessimistic estimate of the noise ceiling due to the small amount of data available in conventional fMRI datasets, the parametric nature of the Monte Carlo estimator results in overly optimistic estimates. For this reason, for real data, we propose a robust procedure to the estimation of the noise ceiling based on bootstraps.Author SummaryEncoding computational models in brain responses measured with fMRI allows testing the algorithmic representations carried out by the neural population within voxels. The accuracy of a model in predicting new responses is used as a measure of the brain validity of this model, but the result of this analysis is determined not only by how precisely the model describes the responses but also by the quality of the data. In this article, we validate existing approaches to estimate the best possible accuracy that any computational model can achieve conditioned to the amount of measurement noise that is present in the experimental data (i.e. the noise ceiling). Additionally we introduce a close form estimation of the noise ceiling that does not require computationally or data expensive procedures. All the methods are compared using simulated and real fMRI data. We draw conclusions over the impact of regularisation procedures and model complexity and make practical recommendations on how to report the results of computational models in neuroimaging.