Affiliation:
1. Department of Computer Science, Tokyo Institute of Technology, Yokohama 226-8502, Japan
Abstract
Emotion recognition is crucial in understanding human affective states with various applications. Electroencephalography (EEG)—a non-invasive neuroimaging technique that captures brain activity—has gained attention in emotion recognition. However, existing EEG-based emotion recognition systems are limited to specific sensory modalities, hindering their applicability. Our study innovates EEG emotion recognition, offering a comprehensive framework for overcoming sensory-focused limits and cross-sensory challenges. We collected cross-sensory emotion EEG data using multimodal emotion simulations (three sensory modalities: audio/visual/audio-visual with two emotion states: pleasure or unpleasure). The proposed framework—filter bank adversarial domain adaptation Riemann method (FBADR)—leverages filter bank techniques and Riemannian tangent space methods for feature extraction from cross-sensory EEG data. Compared with Riemannian methods, filter bank and adversarial domain adaptation could improve average accuracy by 13.68% and 8.36%, respectively. Comparative analysis of classification results proved that the proposed FBADR framework achieved a state-of-the-art cross-sensory emotion recognition performance and reached an average accuracy of 89.01% ± 5.06%. Moreover, the robustness of the proposed methods could ensure high cross-sensory recognition performance under a signal-to-noise ratio (SNR) ≥ 1 dB. Overall, our study contributes to the EEG-based emotion recognition field by providing a comprehensive framework that overcomes limitations of sensory-oriented approaches and successfully tackles the difficulties of cross-sensory situations.
Funder
The Japan Science and Technology Agency (JST) CREST
The Japan Science and Technology Agency (JST) COI-NEXT
The Japan Society for the Promotion of Science (JSPS) KAKENHI
The Japan Science and Technology Agency (JST): The establishment of university fellowships towards the creation of science technology innovation
The Tokyo Institute of Technology (Tokyo Tech) Academy for Convergence of Materials and Informatics
Reference69 articles.
1. Emotion recognition using multi-modal data and machine learning techniques: A tutorial and review;Zhang;Inf. Fusion,2020
2. A review, current challenges, and future possibilities on emotion recognition using machine learning and physiological signals;Bota;IEEE Access,2019
3. Brain-computer interfaces using electrocorticographic signals;Schalk;IEEE Rev. Biomed. Eng.,2011
4. Physical principles of brain—Computer interfaces and their applications for rehabilitation, robotics and control of human brain states;Hramov;Phys. Rep.,2021
5. A survey of affective brain computer interfaces: Principles, state-of-the-art, and challenges;Allison;Brain-Comput. Interfaces,2014