Abstract
ABSTRACTPrediction methods inputting embeddings from protein Language Models (pLMs) have reached or even surpassed state-of-the-art (SOTA) performance on many protein prediction tasks. In natural language processing (NLP) fine-tuning Language Models has become thede factostandard. In contrast, most pLM-based protein predictions do not back-propagate to the pLM. Here, we compared the fine-tuning of three SOTA pLMs (ESM2, ProtT5, Ankh) on eight different tasks. Two results stood out. Firstly, task-specific supervised fine-tuning almost always improved downstream predictions. Secondly, parameter-efficient fine-tuning could reach similar improvements consuming substantially fewer resources. Put simply: always fine-tune pLMs and you will mostly gain. To help you, we provided easy-to-use notebooks for parameter efficient fine-tuning of ProtT5 for per-protein (pooling) and per-residue prediction tasks athttps://github.com/agemagician/ProtTrans/tree/master/Fine-Tuning.
Publisher
Cold Spring Harbor Laboratory
Reference68 articles.
1. Vaswani, A. et al. Attention is all you need. Adv. neural information processing systems 30 (2017).
2. OpenAI. Gpt-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023).
3. Anil, R. et al. Palm 2 technical report. Preprint at https://arxiv.org/abs/2305.10403 (2023).
4. Bubeck, S. et al. Sparks of artificial general intelligence: Early experiments with gpt-4. Preprint at https://arxiv.org/abs/2303.12712 (2023).
5. Liu, Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. Proc. IEEE/CVF international conference on computer vision 10012–10022 (2021).
Cited by
9 articles.
订阅此论文施引文献
订阅此论文施引文献，注册后可以免费订阅5篇论文的施引文献，订阅后可以查看论文全部施引文献