Affiliation:
1. Department of Computer Science, City University of Hong Kong, Hong Kong, Hong Kong, Hong Kong
2. Adobe Research, San Jose, California, United States of America
Abstract
Adjusting the photo color to associate with some design elements is an essential way for a graphic design to effectively deliver its message and make it aesthetically pleasing. However, existing tools and previous works face a dilemma between the ease of use and level of expressiveness. To this end, we introduce an interactive language-based approach for photo recoloring, which provides an intuitive system that can assist both experts and novices on graphic design. Given a graphic design containing a photo that needs to be recolored, our model can predict the source colors and the target regions, and then recolor the target regions with the source colors based on the given language-based instruction. The multi-granularity of the instruction allows diverse user intentions. The proposed novel task faces several unique challenges, including:
1) color accuracy
for recoloring with exactly the same color from the target design element as specified by the user;
2) multi-granularity instructions
for parsing instructions correctly to generate a specific result or multiple plausible ones; and
3) locality
for recoloring in semantically meaningful local regions to preserve original image semantics. To address these challenges, we propose a model called
LangRecol
with two main components: the language-based source color prediction module and the semantic-palette-based photo recoloring module. We also introduce an approach for generating a synthetic graphic design dataset with instructions to enable model training. We evaluate our model via extensive experiments and user studies. We also discuss several practical applications, showing the effectiveness and practicality of our approach. Please find the code and data at https://zhenwwang.github.io/langrecol.
Funder
GRF grant from the Research Grants Council of Hong Kong
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Graphics and Computer-Aided Design
Reference76 articles.
1. Mahmoud Afifi , Abdullah Abuolaim , Mostafa Hussien , Marcus A Brubaker , and Michael S Brown . 2021 a. CAMS: Color-Aware Multi-Style Transfer. arXiv preprint arXiv:2106.13920 (2021). Mahmoud Afifi, Abdullah Abuolaim, Mostafa Hussien, Marcus A Brubaker, and Michael S Brown. 2021a. CAMS: Color-Aware Multi-Style Transfer. arXiv preprint arXiv:2106.13920 (2021).
2. HistoGAN: Controlling Colors of GAN-Generated and Real Images via Color Histograms
3. Pigment-based recoloring of watercolor paintings
4. Unmixing-Based Soft Color Segmentation for Image Manipulation
5. Semantic soft segmentation