CFGPFSR: A Generative Method Combining Facial and GAN Priors for Face Super-Resolution

Abstract

AbstractIn recent years, facial prior has been widely applied to enhance the quality of super-resolution (SR) facial images in face super-resolution (FSR) methods based on deep learning. However, most of the existing facial prior-based FSR methods have insufficient attention to local texture details, which can cause the generated SR facial images with overly smooth and unrealistic texture details, and show obvious artifacts under large magnification. With the help of GAN prior, recent advances can produce excellent results in terms of fidelity and realness. A generative framework for FSR is proposed in this work, which combines GAN and facial prior, termed CFGPFSR. Firstly, we pre-train a face StyleGAN2 and a face parsing network (FPN) that can generate decent parsing maps, in which the proposed CFGPFSR exploits rich and varied priors encapsulated in the face StyleGAN2 (GAN prior) and face parsing maps extracted from the FPN (facial prior) for FSR. Moreover, we introduce the Channel-Split Spatial Feature Transform (CS-SFT) method to further improve FSR performance. GAN and facial priors are introduced into the FSR process through the designed CS-SFT layers so that SR facial images obtain a promising balance between fidelity and realness. Unlike GAN inversion methods which necessitate costly image optimization at runtime, the proposed CFGPFSR can jointly recover facial details by only utilizing one forward pass. Experimental results on synthetic and real images indicate that the proposed CFGPFSR obtains remarkable performance in 16 × SR task, and some of its metrics such as peak signal to noise ratio (PSNR) and structural similarity (SSIM) are higher than that of the comparison methods. Meanwhile, it shows impressive results in reconstructing high-quality facial images.