Extended Reality Telemedicine Collaboration System Using Patient Avatar Based on 3D Body Pose Estimation
Author:
Šarić Matko1, Russo Mladen1ORCID, Kraljević Luka1ORCID, Meter Davor1
Affiliation:
1. Faculty of Electrical Engineering, Mechanical Engineering and Naval Architecture, University of Split, Ruđera Boškovića 32, 21000 Split, Croatia
Abstract
Recent advances in extended reality (XR) technology have opened the possibility of significantly improving telemedicine systems. This is primarily achieved by transferring 3D information about patient state, which is utilized to create more immersive experiences on VR/AR headsets. In this paper, we propose an XR-based telemedicine collaboration system in which the patient is represented as a 3D avatar in an XR space shared by local and remote clinicians. The proposed system consists of an AR client application running on Microsoft HoloLens 2 used by a local clinician, a VR client application running on the HTC vive Pro used by a remote clinician, and a backend part running on the server. The patient is captured by a camera on the AR side, and the 3D body pose estimation is performed on frames from this camera stream to form a 3D patient avatar. Additionally, the AR and VR sides can interact with the patient avatar via virtual hands, and annotations can be performed on a 3D model. The main contribution of our work is the use of 3D body pose estimation for the creation of a 3D patient avatar. In this way, 3D body reconstruction using depth cameras is avoided, which reduces system complexity and hardware and network resources. Another contribution is the novel architecture of the proposed system, where audio and video streaming are realized using WebRTC protocol. The performance evaluation showed that the proposed system ensures high frame rates for both AR and VR client applications, while the processing latency remains at an acceptable level.
Funder
Virtual Telemedicine Assistance
Subject
Electrical and Electronic Engineering,Biochemistry,Instrumentation,Atomic and Molecular Physics, and Optics,Analytical Chemistry
Reference21 articles.
1. Petkova, R., Poulkov, V., Manolova, A., and Tonchev, K. (2022). Challenges in Implementing Low-Latency Holographic-Type Communication Systems. Sensors, 22. 2. Qualcomm Technologies, Inc. (2018). VR and AR Pushing Connectivity Limits, Qualcomm Technologies, Inc. 3. Lin, P.J., Tsai, B.C., and Tsai, Y.W. (2022, January 27–29). Telemedicine System Based on Mixed Reality and Cognitive Speech Service Technologies. Proceedings of the 2022 IEEE 4th Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability (ECBIOS), Tainan, Taiwan. 4. Anton, D., Kurillo, G., Yang, A.Y., and Bajcsy, R. (2017, January 4–6). Augmented telemedicine platform for real-time remote medical consultation. Proceedings of the MultiMedia Modeling: 23rd International Conference, MMM 2017, Reykjavik, Iceland. Proceedings, Part I 23. 5. Carbone, M., Freschi, C., Mascioli, S., Ferrari, V., and Ferrari, M. (2016, January 15–18). A wearable augmented reality platform for telemedicine. Proceedings of the Augmented Reality, Virtual Reality, and Computer Graphics: Third International Conference, AVR 2016, Lecce, Italy. Proceedings, Part II 3.
|
|