Proteins Need Extra Attention: Improving the Predictive Power of Protein Language Models on Mutational Datasets with Hint Tokens

Abstract

ABSTRACTIn this computational study, we introduce “hint token learning,” a novel machine learning approach designed to enhance protein language modeling. This method effectively addresses the unique challenges of protein mutational datasets, characterized by highly similar inputs that may differ by only a single token. Our research highlights the superiority of hint token learning over traditional fine-tuning methods through three distinct case studies. We first developed a highly accurate free energy of folding model using the largest protein stability dataset to date. Then, we applied hint token learning to predict a biophysical attribute, the brightness of green fluorescent protein mutants. In our third case, hint token learning was utilized to assess the impact of mutations on RecA bioactivity. These diverse applications collectively demonstrate the potential of hint token learning for improving protein language modeling across general and specific mutational datasets. To facilitate broader use, we have integrated our protein language models into the HuggingFace ecosystem for downstream, mutational fine-tuning tasks.