Prior Knowledge-guided Hierarchical Action Quality Assessment with 3D Convolution and Attention Mechanism

Abstract

Abstract Recently, there has been a growing interest in the field of computer vision and deep learning regarding a newly emerging problem known as action quality assessment (AQA). However, most researchers still rely on the traditional approach of using models from the video action recognition field. Unfortunately, this approach overlooks crucial features in AQA, such as movement fluency and degree of completion. Alternatively, some researchers have employed the transformer paradigm to capture action details and overall action integrity, but the high computational cost associated with transformers makes them impractical for real-time tasks. Due to the diversity of action types, it is challenging to rely solely on a shared model for quality assessment of various types of actions. To address these issues, we propose a novel network structure for AQA, which is the first to integrate multi-model capabilities through a classification model. Specifically, we utilize a pre-trained I3D model equipped with a self-attention block for classification. This allows us to evaluate various categories of actions using just one model. Furthermore, we introduce self-attention mechanisms and multi-head attention into the traditional convolutional neural network. By systematically replacing the last few layers of the conventional convolutional network, our model gains a greater ability to sense the global coordination of different actions. We have verified the effectiveness of our approach on the AQA-7 dataset. In comparison to other popular models, our model achieves satisfactory performance while maintaining a low computational cost.