Multimodal Emotion Recognition and State Analysis of Classroom Video and Audio Based on Deep Neural Network

Abstract

In the process of learning, learners will express their emotions through a variety of forms, facial expressions and voice are more obvious, which are most easily obtained through computers. In the previous methods, it is mainly based on a single modal, such as expression, speech, text and so on. Due to the diversity of information, the accuracy rate of multimodal recognition is higher than that of single modal recognition. Therefore, this paper proposed a DNN-based multimodal learning emotion analysis method which combines video and speech to detect students’ learning emotion in real time. We use this method to automatically identify learning emotions in primary school English classroom. According to different learning emotions, the PAD emotion scale was used to correspond learning emotions with learning states. Teachers can judge students’ learning state according to the change of students’ learning emotions, so as to adjust teaching methods and strategies in time.