Lightweight Three-Dimensional Pose and Joint Center Estimation Model for Rehabilitation Therapy

Abstract

In this study, we proposed a novel transformer-based model with independent tokens for estimating three-dimensional (3D) human pose and shape from monocular videos, specifically focusing on its application in rehabilitation therapy. The main objective is to recover pixel-aligned rehabilitation-customized 3D human poses and body shapes directly from monocular images or videos, which is a challenging task owing to inherent ambiguity. Existing human pose estimation methods heavily rely on the initialized mean pose and shape as prior estimates and employ parameter regression with iterative error feedback. However, video-based approaches face difficulties capturing joint-level rotational motion and ensuring local temporal consistency despite enhancing single-frame features by modeling the overall changes in the image-level features. To address these limitations, we introduce two types of characterization tokens specifically designed for rehabilitation therapy: joint rotation and camera tokens. These tokens progressively interact with the image features through the transformer layers and encode prior knowledge of human 3D joint rotations (i.e., position information derived from large-scale data). By updating these tokens, we can estimate the SMPL parameters for a given image. Furthermore, we incorporate a temporal model that effectively captures the rotational temporal information of each joint, thereby reducing jitters in local parts. The performance of our method is comparable with those of the current best-performing models. In addition, we present the structural differences among the models to create a pose classification model for rehabilitation. We leveraged ResNet-50 and transformer architectures to achieve a remarkable PA-MPJPE of 49.0 mm for the 3DPW dataset.