Affiliation:
1. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, China
2. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, and Zhongguancun Laboratory, China
3. ARC Lab, Tencent PCG, China
Abstract
Synthesizing photorealistic 4D human head avatars from videos is essential for VR/AR, telepresence, and video game applications. Although existing Neural Radiance Fields (NeRF)-based methods achieve high-fidelity results, the computational expense limits their use in real-time applications. To overcome this limitation, we introduce
BakedAvatar
, a novel representation for real-time neural head avatar synthesis, deployable in a standard polygon rasterization pipeline. Our approach extracts deformable multi-layer meshes from learned isosurfaces of the head and computes expression-, pose-, and view-dependent appearances that can be baked into static textures for efficient rasterization. We thus propose a three-stage pipeline for neural head avatar synthesis, which includes learning continuous deformation, manifold, and radiance fields, extracting layered meshes and textures, and fine-tuning texture details with differential rasterization. Experimental results demonstrate that our representation generates synthesis results of comparable quality to other state-of-the-art methods while significantly reducing the inference time required. We further showcase various head avatar synthesis results from monocular videos, including view synthesis, face reenactment, expression editing, and pose editing, all at interactive frame rates on commodity devices. Source codes and demos are available on our project page.
Funder
National Natural Science Foundation of China
Fundamental Research Funds for the Central Universities
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Graphics and Computer-Aided Design
Reference68 articles.
1. HyperReel: High-Fidelity 6-DoF Video with Ray-Conditioned Sampling
2. SAL: Sign Agnostic Learning of Shapes From Raw Data
3. Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos
4. Valentin Bazarevsky , Yury Kartynnik , Andrey Vakunov , Karthik Raveendran , and Matthias Grundmann . 2019 . Blazeface: Sub-millisecond neural face detection on mobile gpus. arXiv preprint arXiv:1907.05047 (2019). Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. 2019. Blazeface: Sub-millisecond neural face detection on mobile gpus. arXiv preprint arXiv:1907.05047 (2019).
5. Faceware-house: A 3d facial expression database for visual computing;Cao Chen;IEEE Transactions on Visualization and Computer Graphics,2013