Spatio Temporal Joint Distance Maps for Skeleton-Based Action Recognition Using Convolutional Neural Networks

Abstract

Skeleton-based action recognition has become popular with the recent developments in sensor technology and fast pose estimation algorithms. The existing research works have attempted to address the action recognition problem by considering either spatial or temporal dynamics of the actions. But, both the features (spatial and temporal) would contribute to solve the problem. In this paper, we address the action recognition problem using 3D skeleton data by introducing eight Joint Distance Maps, referred to as Spatio Temporal Joint Distance Maps (ST-JDMs), to capture spatio temporal variations from skeleton data for action recognition. Among these, four maps are defined in spatial domain and remaining four are in temporal domain. After construction of ST-JDMs from an action sequence, they are encoded into color images. This representation enables us to fine-tune the Convolutional Neural Network (CNN) for action classification. The empirical results on the two datasets, UTD MHAD and NTU RGB+D, show that ST-JDMs outperforms the other state-of-the-art skeleton-based approaches by achieving recognition accuracies 91.63% and 80.16%, respectively.