Hierarchical Cross-Modal Interaction and Fusion Network Enhanced with Self-Distillation for Emotion Recognition in Conversations

Abstract

Emotion recognition in conversations (ERC), which aims to capture the dynamic changes in emotions during conversations, has recently attracted a huge amount of attention due to its importance in providing engaging and empathetic services. Considering that it is difficult for unimodal ERC approaches to capture emotional shifts in conversations, multimodal ERC research is on the rise. However, this still suffers from the following limitations: (1) failing to fully explore richer multimodal interactions and fusion; (2) failing to dynamically model speaker-dependent context in conversations; and (3) failing to employ model-agnostic techniques to eliminate semantic gaps among different modalities. Therefore, we propose a novel hierarchical cross-modal interaction and fusion network enhanced with self-distillation (HCIFN-SD) for ERC. Specifically, HCIFN-SD first proposes three different mask strategies for extracting speaker-dependent cross-modal conversational context based on the enhanced GRU module. Then, the graph-attention-based multimodal fusion (MF-GAT) module constructs three directed graphs for representing different modality spaces, implements in-depth cross-modal interactions for propagating conversational context, and designs a new GNN layer to address over-smoothing. Finally, self-distillation is employed to transfer knowledge from both hard and soft labels to supervise the training process of each student classifier for eliminating semantic gaps between different modalities and improving the representation quality of multimodal fusion. Extensive experimental results on IEMOCAP and MELD demonstrate that HCIFN-SD is superior to the mainstream state-of-the-art baselines by a significant margin.