Natural Image Reconstruction from fMRI Based on Node–Edge Interaction and Multi–Scale Constraint

Abstract

Reconstructing natural stimulus images using functional magnetic resonance imaging (fMRI) is one of the most challenging problems in brain decoding and is also the crucial component of a brain–computer interface. Previous methods cannot fully exploit the information about interactions among brain regions. In this paper, we propose a natural image reconstruction method based on node–edge interaction and a multi–scale constraint. Inspired by the extensive information interactions in the brain, a novel graph neural network block with node–edge interaction (NEI–GNN block) is presented, which can adequately model the information exchange between brain areas via alternatively updating the nodes and edges. Additionally, to enhance the quality of reconstructed images in terms of both global structure and local detail, we employ a multi–stage reconstruction network that restricts the reconstructed images in a coarse–to–fine manner across multiple scales. Qualitative experiments on the generic object decoding (GOD) dataset demonstrate that the reconstructed images contain accurate structural information and rich texture details. Furthermore, the proposed method surpasses the existing state–of–the–art methods in terms of accuracy in the commonly used n–way evaluation. Our approach achieves 82.00%, 59.40%, 45.20% in n–way mean squared error (MSE) evaluation and 83.50%, 61.80%, 46.00% in n–way structural similarity index measure (SSIM) evaluation, respectively. Our experiments reveal the importance of information interaction among brain areas and also demonstrate the potential for developing visual–decoding brain–computer interfaces.