DeepGene: An Efficient Foundation Model for Genomics based on Pan-genome Graph Transformer

Abstract

ABSTRACTDecoding the language of DNA sequences is a fundamental problem in genome research. Mainstream pre-trained models like DNABERT-2 and Nucleotide Transformer have demonstrated remarkable achievements across a spectrum of DNA analysis tasks. Yet, these models still face the pivotal challenge of (1) genetic language diversity, or the capability to capture genetic variations across individuals or populations in the foundation models; (2) model efficiency, specifically how to enhance performance at scalable costs for large-scale genetic foundational models; (3) length extrapolation, or the ability to accurately interpret sequences ranging from short to long within a unified model framework. In response, we introduce DeepGene, a model leveraging Pan-genome and Minigraph representations to encompass the broad diversity of genetic language. DeepGene employs the rotary position embedding to improve the length extrapolation in various genetic analysis tasks. On the 28 tasks in Genome Understanding Evaluation, DeepGene reaches the top position in 9 tasks, second in 5, and achieves the overall best score. DeepGene outperforms other cutting-edge models for its compact model size and superior efficiency in processing sequences of varying lengths. The datasets and source code of DeepGene are available at GitHub (https://github.com/wds-seu/DeepGene).