Author:
Bai Xuemei,Tan Jiaqi,Hu Hanping,Zhang Chenjie,Gu Dongbing
Abstract
Abstract
In this essay, a model-level fusion technique of multi-modal physiological signals using Multi-Head Attention is studied. A framework that utilizes multi-model physiological signals for the task of emotion classification is proposed. First, the GCRNN model, which combines the Graph Convolutional Network (GCN) and the Long and Short Term Memory (LSTM), captures the unique features of electroencephalogram (EEG) signals. The spatial and temporal information that makes up impulses from the EEG can be captured precisely by such a technique. The CCRNN model, which combines the Convolutional Neural Network (CNN) integrated with the Channel-wise Attention and the LSTM, is used for peripheral physiological signals. The model can extract useful features from peripheral physiological signals and automatically learn to weigh the importance of various channels. Finally, Multi-head Attention is employed to fuse the output of the GCRNN and CCRNN methods. The Multi-head Attention can automatically learn the relevance and importance of different modal signals and weigh them accordingly. Emotion classification is implemented by adding a component of Softmax to map what the model produced to discrete emotion categories. The DEAP dataset was utilized in this study for experimental verification, and the results indicate that the method using multi-modal physiological signal fusion is substantially greater in precision than the technique using simply EEG signals. Additionally, the Multi-head Attention fusion method performs better than previous fusion techniques.
Subject
Computer Science Applications,History,Education
Reference10 articles.
1. A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges [J];Christian;Brain-Computer Interfaces,2014
2. Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals [J];Alzoubi;IEEE Transactions on Affective Computing,2012
3. Emotion Recognition Using Three-Dimensional Feature and Convolutional Neural Network from Multichannel EEG Signals [J];Hao;IEEE Sensors Journal,2020
4. EEG Emotion Recognition Using Dynamical Graph Convolutional Neural Networks [J];Song,2018
5. Investigating Critical Frequency Bands and Channels for EEG-Based Emotion Recognition with Deep Neural Networks [J];Zheng;IEEE Transactions on Autonomous Mental Development,2015