Real-time light-field generation based on the visual hull for the 3D light-field display with free-viewpoint texture mapping

Abstract

Real-time dense view synthesis based on three-dimensional (3D) reconstruction of real scenes is still a challenge for 3D light-field display. It’s time-consuming to reconstruct an entire model, and then the target views are synthesized afterward based on volume rendering. To address this issue, Light-field Visual Hull (LVH) is presented with free-viewpoint texture mapping for 3D light-field display, which can directly produce synthetic images with the 3D reconstruction of real scenes in real-time based on forty free-viewpoint RGB cameras. An end-to-end subpixel calculation procedure of the synthetic image is demonstrated, which defines a rendering ray for each subpixel based on light-field image coding. In the ray propagation process, only the essential spatial point of the target model is located for the corresponding subpixel by projecting the frontmost point of the ray to all the free-viewpoints, and the color of each subpixel is identified in one pass. A dynamic free-viewpoint texture mapping method is proposed to solve the correct graphic texture considering the free-viewpoint cameras. To improve the efficiency, only the visible 3D position and texture that contributes to the synthetic image are calculated based on backward ray tracing rather than computing the entire 3D model and generating all elemental images. In addition, an incremental calibration method by dividing camera groups is proposed to satisfy the accuracy. Experimental results show the validity of our method. All the rendered views are analyzed for justifying the texture mapping method, and the PSNR is improved by an average of 11.88dB. Finally, LVH can achieve a natural and smooth viewing effect at 4K resolution and the frame rate of 25 ∼ 30fps with a large viewing angle.