Augmented reality three-dimensional visualization with multifocus sensing

Abstract

In augmented reality displays, digital information can be integrated with real-world scenes. We present an augmented reality-based approach for three-dimensional optical visualization and depth map retrieval of a scene using multifocus sensing. From a sequence of images captured with different focusing distances, all-in-focus image reconstruction can be performed along with different point of view synthesis. By means of an algorithm that compares the all-in-focus image reconstruction with each image of the z-stack, the depth map of the scene can also be retrieved. Once the three-dimensional reconstructed scene for different points of view along with its depth map is obtained, it can be optically displayed in smart glasses allowing the user to visualize the real three-dimensional scene along with synthesized perspectives of it and provide information such as depth maps of the scene, which are not possible with conventional augmented reality devices. To the best of our knowledge, this is the first report on combining multifocus sensing and three-dimensional visualization and depth retrieval for applications to augmented reality.