Identifying and Monitoring Students’ Classroom Learning Behavior Based on Multisource Information

Abstract

Understanding human activity and behavior, particularly real-time understanding in video feeds, is one of the most active areas of research in Computer Vision (CV) and Artificial Intelligence (AI) nowadays. To advance the topic of integrating learning engagement research with university teaching practice, accurate and efficient assessment, and analysis of students’ classroom learning behavior engagement is very important. The recently proposed classroom behavior recognition algorithms have some limitations, such as the inability to quickly and accurately identify students’ classroom behaviors because they do not consider the motion information of students between consecutive frames. In recent years, action recognition algorithms based on Convolutional Neural Networks (CNN) have improved significantly. To address the limitations of existing algorithms, in this study, a 3D-CNN is selected as a network model for classroom student behavior recognition, which increases information multisourcing and classroom student localization with high accuracy and robustness. For better analysis of human behavior in videos, the 3D convolution extends the 2D convolution to the spatial–temporal domain. In the proposed system, first of all, a real-time picture stream of each student is obtained by combining real-time target detection and tracking. Then, a deep spatiotemporal residual CNN is used to learn the spatiotemporal features of each student’s behavior, so, as to achieve real-time recognition of classroom behaviors for multistudent targets in classroom teaching scenarios. To verify the effectiveness of the proposed model, different experiments are conducted using the labeled classroom behavior dataset. The experimental results demonstrate that the proposed model exhibits better performance in classroom behavior recognition. The accurate recognition of classroom behaviors can assist the teachers and students to understand the classroom learning situation and help to promote the development of smart classroom.