GIGAN: Self‐supervised GAN for generating the invisible using cycle transformation and conditional normalization

Abstract

AbstractObjects in a real scene often occlude each other and inferring a complete appearance from the visible part is an important and challenging task. In this paper, the authors propose a self‐supervised generative adversarial network GIGAN (GAN for generating the invisible), which can generate the complete appearance of objects without labelled invisible part information. The authors build two cycle transformation networks CycleIncomplete (CycleI) and CycleComplete (CycleC) that share parameters to improve the accuracy of mask completion. This design does not require well‐matched training images and can make better use of the limited labelled samples. In addition, the authors propose a conditional normalization module and combine it with the inferred complete mask output. The combination not only enhances the content recovery ability and obtains more realistic outputs, but also improves the efficiency of the generation process. Experimental results show that compared with existing self‐supervised learning models, our method achieves l1 error, mean intersection‐over‐union (mIOU), and Fréchet inception distance (FID) improvements on the COCOA and KINS datasets.