A Comprehensive Evaluation of Self Attention for Detecting Feature Interactions

Abstract

AbstractUsing deep learning in computational biology requires methods that are able to extract meaningful biological information from the trained models. Although deep learning models excel in their prediction performance, interpreting them presents a challenge. Recent work has suggested that self-attention layers can be interpreted to predict cooperativity between binding of transcription factors. We extend this earlier work and demonstrate that the addition of an entropy term to the loss function yields sparser attention values that are both easier to interpret and provide higher precision interpretations. Furthermore, we performed a comprehensive evaluation of the relative performance of different flavors of attention-based transcription factor cooperativity discovery methods, and compared methods that use raw attention scores to the use of attribution over the attention scores, and the earlier DFIM model. We found that the entropy-enhanced attention-based models performed similarly to each other, and exhibited improved accuracy and reduced computation time compared to DFIM.