MRGazer: decoding eye gaze points from functional magnetic resonance imaging in individual space

Abstract

Abstract Objective. Eye-tracking research has proven valuable in understanding numerous cognitive functions. Recently, Frey et al provided an exciting deep learning method for learning eye movements from functional magnetic resonance imaging (fMRI) data. It employed the multi-step co-registration of fMRI into the group template to obtain eyeball signal, and thus required additional templates and was time consuming. To resolve this issue, in this paper, we propose a framework named MRGazer for predicting eye gaze points from fMRI in individual space. Approach. The MRGazer consists of an eyeball extraction module and a residual network-based eye gaze prediction module. Compared to the previous method, the proposed framework skips the fMRI co-registration step, simplifies the processing protocol, and achieves end-to-end eye gaze regression. Main results. The proposed method achieved superior performance in eye fixation regression (Euclidean error, EE = 2.04°) than the co-registration-based method (EE = 2.89°), and delivered objective results within a shorter time (∼0.02 s volume−1) than prior method (∼0.3 s volume−1). Significance. The MRGazer is an efficient, simple, and accurate deep learning framework for predicting eye movement from fMRI data, and can be employed during fMRI scans in psychological and cognitive research. The code is available at https://github.com/ustc-bmec/MRGazer.