Forming an Image that Corresponds to the Natural Perception of the Human Vision in Video See-Through Mixed Reality Systems

Abstract

In this work authors investigated the vergence-accommodation conflict that may appear in mixed reality systems. There exist two main types of the mixed reality systems, one is optical see-through mixed reality systems, other one is video see-through mixed reality systems. One of significant differences between them is how the device user observes the surrounding real world. In case of the optical see-through device, it is observed directly with natural observation conditions, however in case of the video see-through systems the real world is image is observed as an image displayed on the device screen. As a result, when observing the real world the video see-though systems might be affected by the vergence-accommodation conflict. Most of existing video see-through mixed reality systems have two fixed cameras located at the front side of the device to capture the image to be displayed for each of the user eyes. However, in this case the vergence conflict appears due to the disagreement between the fixed direction of the external cameras of the mixed reality device with the constantly changing direction of the human eyes. In the current work authors propose the special image processing methods to correct this disagreement and recover the image with correct vergence basing on the RGB images from cameras and real-world depth estimation. Authors considered different number and placement of the front cameras including the solutions using single RGBD camera and multiple RGB cameras. The simulation results are presented. The pros and cons of proposed recovering methods are shown.