Multimodal Pretraining for Unsupervised Protein Representation Learning

Abstract

In this paper, we introduce a framework of symmetry-preserving multimodal pretraining to learn a unified representation of proteins in an unsupervised manner, encompassing both primary and tertiary structures. Our approach involves proposing specific pretraining methods for sequences, graphs, and 3D point clouds associated with each protein structure, leveraging the power of large language models and generative models. We present a novel way to combining representations from multiple sources of information into a single global representation for proteins. We carefully analyze the performance of our framework in the pretraining tasks. For the fine-tuning tasks, our experiments have shown that our new multimodal representation can achieve competitive results in protein-ligand binding affinity prediction, protein fold classification, enzyme identification and mutation stability prediction. We expect that this work will accelerate future research in proteins. Our source code in PyTorch deep learning framework is publicly available athttps://github.com/HySonLab/Protein_Pretrain.