Occlusion relationship reasoning with a feature separation and interaction network

Abstract

AbstractOcclusion relationship reasoning aims to locate where an object occludes others and estimate the depth order of these objects in three-dimensional (3D) space from a two-dimensional (2D) image. The former sub-task demands both the accurate location and the semantic indication of the objects, while the latter sub-task needs the depth order among the objects. Although several insightful studies have been proposed, a key characteristic of occlusion relationship reasoning, i.e., the specialty and complementarity between occlusion boundary detection and occlusion orientation estimation, is rarely discussed. To verify this claim, in this paper, we integrate these properties into a unified end-to-end trainable network, namely the feature separation and interaction network (FSINet). It contains a shared encoder-decoder structure to learn the complementary property between the two sub-tasks, and two separated paths to learn specialized properties of the two sub-tasks. Concretely, the occlusion boundary path contains an image-level cue extractor to capture rich location information of the boundary, a detail-perceived semantic feature extractor, and a contextual correlation extractor to acquire refined semantic features of objects. In addition, a dual-flow cross detector has been customized to alleviate false-positive and false-negative boundaries. For the occlusion orientation estimation path, a scene context learner has been designed to capture the depth order cue around the boundary. In addition, two stripe convolutions are built to judge the depth order between objects. The shared decoder supplies the feature interaction, which plays a key role in exploiting the complementarity of the two paths. Extensive experimental results on the PIOD and BSDS ownership datasets reveal the superior performance of FSINet over state-of-the-art alternatives. Additionally, abundant ablation studies are offered to demonstrate the effectiveness of our design.