Integrating Complementary Appearance, Posture and Motion Cues for RGB-D Action Recognition

Abstract

This work presents a novel approach to multimodal human action recognition by jointly using visual RGB and depth (including skeleton joint positions) data captured from depth camera. For the depth feature extraction, Local Surface Geometric Feature (LSGF) is adopted to capture the geometric appearance and postures cues. Simultaneously, the improved dense trajectory feature (IDT) is extracted from RGB modality to jointly characterize the motion, visual appearance and trajectory shape information. These features from different modalities are complementary to each other. Then a two-stage integration scheme is proposed, which incorporates the probability weights of each classifier for action recognition. The proposed approach is evaluated on four publicly available human action databases: NJUST RGB-D Action, MSR-ActionPairs, MSR-DailyAct3D, and UTD-MHAD. Experimental results demonstrate that the proposed approach outperforms or is comparable to the state-of-the-art methods.