TRACL: Temporal reconstruction and adaptive consistency loss for semi‐supervised video semantic segmentation

Abstract

AbstractWhile existing supervised semantic segmentation methods have shown significant performance improvements, they heavily rely on large‐scale pixel‐level annotated data. To reduce this dependence, recent research has proposed semi‐supervised learning‐based methods that have achieved great success. However, almost all these works are mainly dedicated to image semantic segmentation, while semi‐supervised video semantic segmentation (SVSS) has been barely explored. Due to the significant difference between video data and image, simply adapting semi‐supervised image semantic segmentation approaches to SVSS may neglect the inherent temporal correlations in video frames. This paper presents a novel method (named TRACL) with temporal reconstruction (TR) and adaptive consistency loss (ACL) for SVSS, aiming to fully utilize the temporal relations of internal frames in video clip. The authors’ TR method implements the reconstruction from the feature and output levels to narrow the distribution gap between internal video frames. Specifically, considering the underlying data distribution, the authors construct Gaussian models for each category, and use probability density function to obtain the similarity between different feature maps for temporal feature reconstruction. The authors’ ACL can adaptively select two pixel‐wise consistency loss including Flow Consistency Loss and Reconstruction Consistency Loss, providing stronger supervision signals for unlabelled frames during model training. Additionally, the authors extend their method to unlabelled video for more training data by employing mean‐teacher structure. Extensive experiments on three datasets including Cityscapes, Camvid and VSPW demonstrate that the authors’ proposed method outperforms previous state‐of‐the‐art methods.