Enhancing text classification with attention matrices based on BERT

Abstract

SummaryText classification is a critical task in the field of natural language processing. While pre‐trained language models like BERT have made significant strides in improving performance in this area, the distinctive dependency information that is present in text has not been fully exploited. Besides, BERT mostly captures phrase‐level information in lower layers, which becomes progressively weaker with the increasing depth of layers. To address these limitations, our work focuses on enhancing text classification through the incorporation of Attention Matrices, particularly in the fine‐tuning process of pre‐trained models like BERT. Our approach, named AM‐BERT, leverages learned dependency relationships as external knowledge to enhance the pre‐trained model by generating attention matrices. In addition, we introduce a new learning strategy that enables the model to retain learned phrase‐level structure information. Extensive experiments and detailed analysis on multiple benchmark datasets demonstrate the effectiveness of our approach in text classification tasks. Furthermore, we show that AM‐BERT achieves stable performance improvements also in named entity recognition tasks.