Fuzzy speech emotion recognition considering semantic awareness

Abstract

Speech emotion recognition is of great significance in the industry such as social robots, health care, and intelligent education. Due to the obscurity of emotional expression in speech, most works on speech emotion recognition (SER) ignore the consistency of speech emotion recognition, leading to fuzzy expression and low accuracy in emotional recognition. In this paper, we propose a semantic aware speech emotion recognition model to alleviate this issue. Specifically, a speech feature extraction module based on CNN and Transformer is designed to extract local and global information from the speech. Moreover, a semantic embedding support module is proposed to use text semantic information as auxiliary information to assist the model in extracting emotional features of speech, and can effectively overcome the problem of low recognition rate caused by emotional ambiguity. In addition, the model uses a key-value pair attention mechanism to fuse the features, which makes the fusion of speech and text features preferable. In experiments on two benchmark corpora IEMOCAP and EMO-DB, the recognition rates of 74.3% and 72.5% were obtained under respectively, which show that the proposed model can significantly improve the accuracy of emotion recognition.