Affiliation:
1. MIT, USA and Google, USA
2. ETH Zurich, Switzerland and Google, Switzerland
3. Google, USA
4. Google, United Kingdom
5. MIT, USA
6. Google, Switzerland
Abstract
3D rendering of dynamic face captures is a challenging problem, and it demands improvements on several fronts---photorealism, efficiency, compatibility, and configurability. We present a novel representation that enables high-quality volumetric rendering of an actor's dynamic facial performances with minimal compute and memory footprint. It runs natively on commodity graphics soft- and hardware, and allows for a graceful trade-off between quality and efficiency. Our method utilizes recent advances in neural rendering, particularly learning discrete radiance manifolds to sparsely sample the scene to model volumetric effects. We achieve efficient modeling by learning a single set of manifolds for the entire dynamic sequence, while implicitly modeling appearance changes as temporal canonical texture. We export a single layered mesh and view-independent RGBA texture video that is compatible with legacy graphics renderers without additional ML integration. We demonstrate our method by rendering dynamic face captures of real actors in a game engine, at comparable photorealism to state-of-the-art neural rendering techniques at previously unseen frame rates.
Publisher
Association for Computing Machinery (ACM)
Reference69 articles.
1. SAL: Sign Agnostic Learning of Shapes From Raw Data
2. Learning Personalized High Quality Volumetric Head Avatars from Monocular RGB Videos
3. Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. 2022. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. CVPR (2022).
4. Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields
5. Alexander W. Bergman Petr Kellnhofer Wang Yifan Eric R. Chan David B. Lindell and Gordon Wetzstein. 2023. Generative Neural Articulated Radiance Fields. arXiv:2206.14314 [cs.CV]