Synchronous mixed reality (SMR): A personalized virtual‐real fusion framework with high immersion and effective interaction

Abstract

AbstractVirtual‐real fusion (VRF) technology plays a crucial role in the meta‐universe by bridging the gap between virtual environments (VEs) and physical environments (PEs). However, the current VRF system has the problem of single function and fixed integration, limiting the popularization and application of VRF technology. Therefore, this paper proposes a framework named synchronous mixed reality (SMR) to personalize the VRF system and maintain the balance between interaction efficiency and environmental immersion. We combine an instance segmentation algorithm with an interaction prediction algorithm to achieve this balance. To evaluate the effectiveness of the SMR framework, we design three scenarios based on the interaction properties of physical objects and measure environmental immersion, systematic interaction efficiency, and user experience. Our results demonstrate that the SMR framework meets diverse needs in various PEs while balancing immersion and interaction efficiency. Additionally, our framework significantly reduces collisions and negative emotions experienced by users in VEs. We anticipate that this framework will serve as a guide for constructing virtual‐real fusion systems in the future.