Sketch Video Synthesis

Abstract

AbstractUnderstanding semantic intricacies and high‐level concepts is essential in image sketch generation, and this challenge becomes even more formidable when applied to the domain of videos. To address this, we propose a novel optimization‐based framework for sketching videos represented by the frame‐wise Bézier Curves. In detail, we first propose a cross‐frame stroke initialization approach to warm up the location and the width of each curve. Then, we optimize the locations of these curves by utilizing a semantic loss based on CLIP features and a newly designed consistency loss using the self‐decomposed 2D atlas network. Built upon these design elements, the resulting sketch video showcases notable visual abstraction and temporal coherence. Furthermore, by transforming a video into vector lines through the sketching process, our method unlocks applications in sketch‐based video editing and video doodling, enabled through video composition.