ESMBind and QBind: LoRA, QLoRA, and ESM-2 for Predicting Binding Sites and Post Translational Modification

Abstract

AbstractIn this study we discuss the viability of applying protein language models to the problem of predicting bindings sites of protein sequences from single sequences alone using Low Rank Adaptation (LoRA) and Quantized Low Rank Adaptation (QLoRA). No Multiple Sequence Alignment (MSA) or structural information for the proteins was used. Moreover, using LoRA and QLoRA shows improved performance over vanilla full finetuning, and significantly helps in mitigating overfitting. Also, due to the efficiency of LoRA and QLoRA, we are able to train the larger ESM-2 models on modest hardware, making the method very attractive and accessible. We also note that this technique serves as an important regularization technique and serves to improve generalization of models on unseen data.