Joint Structural-Functional Brain Graph Transformer

Abstract

Multimodal brain graph transformers have become one of the foundational architectures of graph foundation models for brain science, relying on multimodal brain network fusion. However, most current multimodal brain network fusion methods primarily focus on modality-specific information fusion. The interplays within structural-functional brain networks are often ignored. Therefore, they fail to acquire essential coupling information, which is crucial for obtaining robust joint brain network representations. This oversight inevitably limits the effectiveness and generalization of these representations in various downstream tasks. To this end, we propose a novel joint structural-functional brain graph transformer model (namely sfBGT). Technically, we design a cross-network assortativity quantification mechanism to enable structural-functional brain network coupling, thus capturing the interplays of brain structure and function. We then employ a multimodal graph transformer to effectively learn joint representations of structural-functional brain networks along with their coupling relation representations. Experimental results on three real-world datasets demonstrate the superiority of sfBGT over state-of-the-art baselines.