Accurate 3D reconstruction of single-frame speckle-encoded textureless surfaces based on densely connected stereo matching network

Abstract

Speckle projection profilometry (SPP) determines the global correspondence between stereo images by speckle pattern(s) projection in three-dimensional (3D) vision. However, it is extremely challenging for traditional algorithms to achieve a satisfactory 3D reconstruction accuracy generally via single-frame speckle pattern, which heavily constraints the application in dynamic 3D imaging. Recently some deep learning (DL) based methods have made process in this issue but there exist deficiencies in feature extraction, leading to a limited improvement in accuracy. In this paper, we propose a stereo matching network called Densely Connected Stereo Matching (DCSM) Network that requires only single-frame speckle pattern as input, adopts densely connected feature extraction and incorporates attention weight volume construction. The densely connected multi-scale feature extraction module we constructed in DCSM Network has a positive effect on the combination of global and local information and inhibition of information loss. We also establish a real measurement system and its digital twin through Blender to obtain rich speckle data under SPP framework. Meanwhile, we introduce Fringe Projection Profilometry (FPP) to obtain phase information to assist in generating high-precision disparity as Ground Truth (GT). Experiments with different types of models and models with various perspectives are implemented to prove the effectiveness and generalization of the proposed network compared with classic and the latest DL-based algorithms. Finally, the 0.5-Pixel-Error of our method in the disparity maps is as low as 4.81%, and the accuracy is verified to be improved by up to 33.4%. As for the cloud point, our method has a reduction of 18%∼30% compared with other network-based methods.