Emotion Recognition Model Based on Multimodal Decision Fusion

Abstract

Abstract In the process of human social activities and daily communication, speech, text and facial expressions are considered as the main channels to convey human emotions. In this paper, a fusion method of multi-modal emotion recognition based on speech, text and motion is proposed. In the speech emotion recognition (SER), a depth wavefield extrapolation - improved wave physics model (DWE-WPM) is designed. In order to simulate the information mining process of LSTM, a user-defined feature extraction scheme is used to reconstruct the wave and inject it into DWE-WPM. In the text emotion recognition (TER), the transformer model with multi attention mechanism is used to recognize the text emotion combined. In the motion emotion recognition (MER), the sequential features of facial expression and hand action are extracted in groups. Combined with the bidirectional three-layer LSTM model with attention mechanism, a joint model of four channels is designed. Experimental results show that the proposed method has high recognition accuracy in multi-modal, and the accuracy is improved by 9% in the interactive emotional dynamic motion capture (IEMOCAP) corpus.