Author:
Yang Wei,Liu Chun,Li Zheng
Abstract
AbstractPretrained large-scale protein language models, such as ESM-1b and ProtTrans, are becoming the fundamental infrastructure for various protein-related biological modeling tasks. Existing works use mainly pretrained protein language models in feature extraction. However, the knowledge contained in the embedding features directly extracted from a pretrained model is task-agnostic. To obtain task-specific feature representations, a reasonable approach is to fine-tune a pretrained model based on labeled datasets from downstream tasks. To this end, we investigate the fine-tuning of a given pretrained protein language model for protein secondary structure prediction tasks. Specifically, we propose a novel end-to-end protein secondary structure prediction framework involving the lightweight fine-tuning of a pretrained model. The framework first introduces a few new parameters for each transformer block in the pretrained model, then updates only the newly introduced parameters, and then keeps the original pretrained parameters fixed during training. Extensive experiments on seven test sets, namely, CASP12, CASP13, CASP14, CB433, CB634, TEST2016, and TEST2018, show that the proposed framework outperforms existing predictors and achieves new state-of-the-art prediction performance. Furthermore, we also experimentally demonstrate that lightweight fine-tuning significantly outperforms full model fine-tuning and feature extraction in enabling models to predict secondary structures. Further analysis indicates that only a few top transformer blocks need to introduce new parameters, while skipping many lower transformer blocks has little impact on the prediction accuracy of secondary structures.
Publisher
Cold Spring Harbor Laboratory
Reference55 articles.
1. Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
2. Towards designing modular recurrent neural networks in learning protein secondary structures;Expert Systems with Applications,2012
3. Exploiting the past and the future in protein secondary structure prediction
4. Rcsb protein data bank: powerful new tools for exploring 3d structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology, bioengineering and energy sciences;Nucleic Acids Research,2020
5. Busia, A. , Jaitly, N. , 2017. Next-step conditioned deep convolutional neural networks improve protein secondary structure prediction. ArXiv abs/1702.03865.
Cited by
2 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献