A Survey on Multi-modal Emotion Detection Techniques

Abstract

Abstract The utilization of emotion detection and recognition technologies has revolution- ized human-computer interactions in various fields such as sentiment analysis, health monitoring, education, and automotive interfaces. Previously, traditional systems relied on single-channel affect sensing, which limited their ability to cap- ture the complexity of human emotions. However, humans naturally combine multiple cues such as facial expressions, speech, gestures, and contextual factors when expressing their emotions. As a result, there has been a growing inter- est in multi-modal emotion frameworks that integrate different sensory streams to obtain more comprehensive emotion assessments. These holistic perspectives allow for the capture of nuanced affective information that would otherwise be difficult to represent. In this survey paper, we delve into the latest advancements in emotion recognition systems, examining fusion techniques, feature engineer- ing methods, and classification architectures that leverage inputs from various modalities such as vision, audio, and text. Our focus is to showcase innova- tive interventions throughout the entire pipeline, from preprocessing raw signals to predicting emotion labels, in order to enable robust multi-modal analysis. Through detailed theoretical discussions and practical case studies, this paper aims to inspire further research by providing insights into the current state-of- the-art, highlighting open challenges, and exploring promising avenues in emotion detection through cross-modal learning.