End-to-end information fusion method for transformer-based stereo matching

Abstract

Abstract In stereo matching, the application of transformers can overcome the limitations of disparity range and capture long-range matching information. However, the lack of cross-epipolar context information often leads to numerous mismatches, especially in low-texture regions. An end-to-end information fusion stereo matching method is proposed to address this issue. In the proposed method, a feature extraction method that combines dense connections and a residual block is proposed. Global and local semantic information can be effectively fused by incorporating dense connections among multiscale feature maps. Additionally, the inclusion of a residual block helps extract more representative feature maps. The idea of criss-cross attention is introduced in the transformer implicit matching process. Criss-cross attention enables the capture of cross-epipolar context information by combining horizontal and vertical attention mechanisms. This method improves the matching accuracy from the perspective of multi-path information fusion. According to the matching results, the disparity regression layer and the context adjustment layer are used to generate the initial and final disparity maps, respectively. The proposed method is evaluated on the Scene Flow, KITTI 2012, and Middlebury 2014 datasets. Experimental results indicate that the proposed method effectively enhances matching accuracy. Moreover, the proposed method exhibits strong generalization ability, allowing for direct application to synthetic, real outdoor, and real indoor scene images.