Unsupervised video object segmentation: an affinity and edge learning approach

Abstract

AbstractThis paper presents a new approach, called TMNet, to solve unsupervised video object segmentation (UVOS) problem. The UVOS is still a challenging problem as prior methods suffer from issues like generalization errors in unseen test videos, over reliance on optic flow, and capturing fine details at object boundaries. These issues make the UVOS an ill-defined problem, particularly in presence of multiple objects. Our focus is to constrain the problem and improve the segmentation results by fusion of multiple available cues such as appearance and motion, as well as image and flow edges. To constrain the problem, instead of predicting segmentation directly, we predict affinities between neighbouring pixels for being part of the same object and cluster those to obtain category agnostic segmentation. To further improve the segmentation, we fuse multiple-sources of information through a novel Temporal Motion Attention (TMA) module that uses neural attention to learn powerful spatio-temporal features. In addition, we also design an edge refinement module (using image and optic flow edges) to refine and improve the accuracy of object segmentation boundaries. The overall framework is capable of segmenting and finding accurate objects’ boundaries without any heuristic post processing. This enables the method to be used for unseen videos. Experimental results on challenging DAVIS16 and multi object DAVIS17 datasets show that our proposed TMNet performs favorably compared to the state-of-the-art methods without post processing.