Action Recognition Using Form and Motion Modalities

Abstract

Action recognition has attracted increasing interest in computer vision due to its potential applications in many vision systems. One of the main challenges in action recognition is to extract powerful features from videos. Most existing approaches exploit either hand-crafted techniques or learning-based methods to extract features from videos. However, these methods mainly focus on extracting the dynamic motion features, which ignore the static form features. Therefore, these methods cannot fully capture the underlying information in videos accurately. In this article, we propose a novel feature representation method for action recognition, which exploits hierarchical sparse coding to learn the underlying features from videos. The learned features characterize the form and motion simultaneously and therefore provide more accurate and complete feature representation. The learned form and motion features are considered as two modalities, which are used to represent both the static and motion features. These modalities are further encoded into a global representation via a pairwise dictionary learning and then fed to an SVM classifier for action classification. Experimental results on several challenging datasets validate that the proposed method is superior to several state-of-the-art methods.