Abstract
AbstractIn this work, we explore the creation of self-avatars through video pass-through in mixed reality (MR) applications. We present our end-to-end system, including custom MR video pass-through implementation on a commercial head-mounted display (HMD), our deep learning-based real-time egocentric body segmentation algorithm, and our optimized offloading architecture, to communicate the segmentation server with the HMD. To validate this technology, we designed an immersive VR experience where the user has to walk through a narrow tile path over an active volcano crater. The study was performed under three-body representation conditions: virtual hands, video pass-through with color-based full-body segmentation, and video pass-through with deep learning full-body segmentation. This immersive experience was carried out by 30 women and 28 men. To the best of our knowledge, this is the first user study focused on evaluating video-based self-avatars to represent the user in a MR scene. Results showed no significant differences between the different body representations in terms of presence, with moderate improvements in some Embodiment components between the virtual hands and full-body representations. Visual Quality results showed better results from the deep-learning algorithms in terms of the whole body perception and overall segmentation quality. In this study, we provide some discussion regarding the use of video-based self-avatars and some reflections on the evaluation methodology. The proposed E2E solution is in the boundary of the state-of-the-art, so there is still room for improvement before it reaches maturity. However, this solution serves as a crucial starting point for MR applications where users can feel immersed and interact with their own bodies.
Funder
Marie Skłodowska-Curie ETN TeamUp5G
Publisher
Springer Science and Business Media LLC
Subject
Computer Graphics and Computer-Aided Design,Human-Computer Interaction,Software
Reference65 articles.
1. Alaee G, Deasi AP, Pena-Castillo L et al. (2018) A user study on augmented virtuality using depth sensing cameras for near-range awareness in immersive vr. In: IEEE VR’s 4th workshop on everyday virtual reality (WEVR 2018), p 3
2. Argelaguet F, Hoyet L, Trico M et al. (2016) The role of interaction in virtual embodiment: effects of the virtual hand representation. In: Proceedings of IEEE VR, pp 3–10
3. Arora N, Suomalainen M, Pouke M et al. (2022) Augmenting immersive telepresence experience with a virtual body. arXiv:2202.00900
4. Bazarevsky V, Grishchenko I, Raveendran K et al. (2020) Blazepose: on-device real-time body pose tracking. arXiv:2006.10204
5. Bhargava A, Venkatakrishnan R, Venkatakrishnan R et al. (2021) Did I hit the door effects of self-avatars and calibration in a person-plus-virtual-object system on perceived frontal passability in vr. IEEE Trans Vis Comput Graph 28:4198–4210
Cited by
3 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献