Real-Time Speech Emotion Recognition Using Deep Learning and Data Augmentation

Abstract

Abstract In human-human interactions, detecting emotions is often easy as it can be perceived through facial expressions, body gestures, or speech. However, in human-machine interactions, detecting human emotion can be a challenge. To improve this interaction, the term 'speech emotion recognition' has emerged, with the goal of recognizing emotions solely through vocal intonation. In this work, we propose a speech emotion recognition system based on deep learning approaches and two efficient data augmentation techniques (noise addition and spectrogram shifting). To evaluate the proposed system, we used three different datasets: TESS, EmoDB, and RAVDESS. We employe several algorithms such as Mel Frequency Cepstral Coefficients (MFCC), Zero Crossing Rate (ZCR), Mel spectrograms, Root Mean Square Value (RMS), and chroma to select the most appropriate vocal features that represent speech emotions. To develop our speech emotion recognition system, we use three different deep learning models, including MultiLayer Perceptron (MLP), Convolutional Neural Network (CNN), and a hybrid model that combines CNN with Bidirectional Long-Short Term Memory (Bi-LSTM). By exploring these different approaches, we were able to identify the most effective model for accurately identifying emotional states from speech signals in real-time situation. Overall, our work demonstrates the effectiveness of the proposed deep learning model, specifically based on CNN+BiLSTM, and the used two data augmentation techniques for the proposed real-time speech emotion recognition.