Telerehabilitation System Based on OpenPose and 3D Reconstruction with Monocular Camera

Abstract

Owing to aging populations, the number of elderly people with limb dysfunction affecting their daily lives will continue to increase. These populations have a great need for rehabilitation training to restore limb functions. However, the current numbers of rehabilitation hospitals and doctors are limited. Moreover, people often cannot go to a hospital owing to external conditions (e.g., the impacts of COVID-19). Thus, an urgent need exists for telerehabilitation system for allowing patients to have training at home. The purpose of this study is to develop an easy-to-use system for allowing target users to experience rehabilitation training at home and to remotely receive real-time guidance from doctors. The proposed system only needs a monocular camera to capture 3D motions. First, the 2D key joints of the human body are detected; then, a simple baseline network is used to reconstruct 3D key joints from the 2D key joints. The 2D detection only has an average angle error of 1.7% compared to that of a professional motion capture system. In addition, the 3D reconstruction has a mean per-joint position error of only 67.9 mm compared to the real coordinates. After acquiring the user’s 3D motions, the system synchronizes the 3D motions to a virtual human model in Unity, providing the user with a more intuitive and interactive experience. Generally, many telerehabilitation systems require professional motion capture cameras and wearable equipment, and the training target is a single body part. In contrast, the proposed system is low-cost and easier to use and only requires a monocular camera and computer to achieve real-time and intuitive telerehabilitation (even though the training target is the entire body). Furthermore, the system provides a similarity evaluation of the motions based on the dynamic time warping; this can provide more accurate and direct feedback to users. In addition, a series of evaluation experiments verify the system’s usability, convenience, feasibility, and accuracy, with the ultimate conclusion that the system can be used in practical rehabilitation applications.