Scaffold-Lab: Critical Evaluation and Ranking of Protein Backbone Generation Methods in A Unified Framework

Abstract

AbstractDe novoprotein design has undergone a rapid development in recent years, especially for backbone generation, which stands out as more challenging yet valuable, offering the ability to design novel protein folds with fewer constraints. However, a comprehensive delineation of its potential for practical application in protein engineering remains lacking, as does a standardized evaluation framework to accurately assess the diverse methodologies within this field. Here, we proposed Scaffold-Lab benchmark focusing on evaluating unconditional generation across metrics like designability, novelty, diversity, efficiency and structural properties. We also extrapolated our benchmark to include the motif-scaffolding problem, demonstrating the utility of these conditional generation models. Our findings reveal thatFrameFlowandRFdiffusionin unconditional generation along withRfdiffusionand GPDL in conditional generation showcased the most outstanding performances. Furthermore, we described a systematic study to investigate conditional generation and applied it to the motif-scaffolding task, offering a novel perspective for the analysis and development of conditional protein design methods. All data and scripts will be available athttps://github.com/Immortals-33/Scaffold-Lab.