Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing-Reference-Cited by-同舟云学术

Cross-Attention and Seamless Replacement of Latent Prompts for High-Definition Image-Driven Video Editing

Published:2023-12-19 Issue:1 Volume:13 Page:7
ISSN:2079-9292
Container-title:Electronics
language:en
Short-container-title:Electronics

Author:

Zhao Liangbing¹,Zhang Zicheng²,Nie Xuecheng³,Liu Luoqi³,Liu Si⁴

Affiliation:

1. School of Computer Science and Engineering, Beihang University, Beijing 100191, China

2. School of Mathematical Sciences, University of Chinese Academy of Sciences, Beijing 100049, China

3. Meitu Inc., 7th Floor, Block B/C, Yousheng Building, 28 Chengfu Road, Haidian District, Beijing 100083, China

4. Institute of Artificial Intelligence, Beihang University, Beijing 100191, China

Abstract

Recently, text-driven video editing has received increasing attention due to the surprising success of the text-to-image model in improving video quality. However, video editing based on the text prompt is facing huge challenges in achieving precise and controllable editing. Herein, we propose Latent prompt Image-driven Video Editing (LIVE) with a precise and controllable video editing function. The important innovation of LIVE is to utilize the latent codes from reference images as latent prompts to rapidly enrich visual details. The novel latent prompt mechanism endows two powerful capabilities for LIVE: one is a comprehensively interactive ability between video frame and latent prompt in the spatial and temporal dimensions, achieved by revisiting and enhancing cross-attention, and the other is the efficient expression ability of training continuous input videos and images within the diffusion space by fine-tuning various components such as latent prompts, textual embeddings, and LDM parameters. Therefore, LIVE can efficiently generate various edited videos with visual consistency by seamlessly replacing the objects in each frame with user-specified targets. The high-definition experimental results from real-world videos not only confirmed the effectiveness of LIVE but also demonstrated important potential application prospects of LIVE in image-driven video editing.

Funder

National Natural Science Foundation of China

Publisher

MDPI AG

Subject

Electrical and Electronic Engineering,Computer Networks and Communications,Hardware and Architecture,Signal Processing,Control and Systems Engineering

Link

https://www.mdpi.com/2079-9292/13/1/7/pdf

Reference44 articles.

1. Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D.J. (2022). Video diffusion models. arXiv.

2. Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., and Gafni, O. (2022). Make-a-video: Text-to-video generation without text-video data. arXiv.

3. Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., and Fleet, D.J. (2022). Imagen video: High-definition video generation with diffusion models. arXiv.

4. Harvey, W., Naderiparizi, S., Masrani, V., Weilbach, C., and Wood, F. (2022). Flexible diffusion modeling of long videos. arXiv.

5. Molad, E., Horwitz, E., Valevski, D., Rav, A.A., Matias, Y., Pritch, Y., Leviathan, Y., and Yedid, H. (2023). Dreamix: Video diffusion models are general video editors. arXiv.