Digital Twin for Digital Health: Body Joint Modeling and 3D Pose Reconstruction

Abstract

Body joint modeling and human pose reconstruction provide precise motion and quantitative geometric information about human dynamics. The rich motion information obtained from human pose estimation plays important roles in a wide range of digital twin and connected health applications. However, current related researches have difficulties in extracting the joints’ spatial-temporal correlations from different levels. This is due to the poses being at various complexities in moving various joints differently. Hence, the typical conventional transformer method is non-adaptable and barely meets the aforementioned requirement. In this paper, we propose the B ody J oint I nteractive trans Former s (BJIFormer) to extract the multi-level joints’ spatial-temporal information. The design enables the model to learn the inner joints’ correlation inside the body parts across frames and propagate the extracted information across the body parts with shared joints. The multi-level body joint interactive scheme has greater efficiency improvement by restricting the self-attention computation to partial body parts and connecting each body part by torso. The proposed interactive approach explores the spatial-temporal correlation following the hierarchical paradigm and effectively estimates and reconstructs 3D human poses.