Two‐Step Training: Adjustable Sketch Colourization via Reference Image and Text Tag

Abstract

AbstractAutomatic sketch colourization is a highly interestinged topic in the image‐generation field. However, due to the absence of texture in sketch images and the lack of training data, existing reference‐based methods are ineffective in generating visually pleasant results and cannot edit the colours using text tags. Thus, this paper presents a conditional generative adversarial network (cGAN)‐based architecture with a pre‐trained convolutional neural network (CNN), reference‐based channel‐wise attention (RBCA) and self‐adaptive multi‐layer perceptron (MLP) to tackle this problem. We propose two‐step training and spatial latent manipulation to achieve high‐quality and colour‐adjustable results using reference images and text tags. The superiority of our approach in reference‐based colourization is demonstrated through qualitative/quantitative comparisons and user studies with existing network‐based methods. We also validate the controllability of the proposed model and discuss the details of our latent manipulation on the basis of experimental results of multi‐label manipulation.