Self-Adaptive Clothing Mapping Based Virtual Try-on

Abstract

VTON (Virtual Try-ON), as an innovative visual application in e-commerce scenarios with great commercial value, has been widely studied in recent years. Due to its better robustness and realistic effect, deformation-synthesize-based VTON has become the dominant approach in this field. Existing clothing deformation techniques optimize the mapping relations between the original clothing image and the ground truth (GT) image of the worn clothing. However, there are color differences between the original and GT clothing images caused by lighting, warping, and occlusion. The color differences may lead to misaligned clothing mapping by only minimizing the cost of pixel value difference. Another drawback is that taking the parsing prediction as GT will bring alignment remnant, rooting in the processing order of parsing and deformation. Aiming above two drawbacks, we put forward SAME-VTON (Self-Adaptive clothing Mapping basEd Virtual Try-ON) for achieving realistic virtual try-on results. The core of SAME-VTON is the self-adaptive clothing mapping technique, composed of two parts: a color-adaptive clothing mapping module and a parsing-adaptive prediction process. In the color-adaptive clothing mapping module, we map each pixel of the target clothing with a combination of multiple pixel values from the original clothing image, which considers both the position and color changes. Furthermore, different combination weights are learned to increase the diversity of color mapping. In the parsing-adaptive prediction process, the color-adaptive clothing mapping module is adopted to deform clothing first, then the human parsing result is predicted under the reference of the deformed clothing, which can avoid alignment remnant. Extensive experiments demonstrate that the proposed SAME-VTON with the self-adaptive clothing mapping technique can achieve optimal mapping in the case of large color differences and obtain superior results compared with existing deformation-synthesize-based VTON.