Affiliation:
1. Tsinghua University, Beijing, China
2. NNKosmos Technology, Beijing, China
Abstract
We present AvatarReX, a new method for learning NeRF-based full-body avatars from video data. The learnt avatar not only provides expressive control of the body, hands and the face together, but also supports real-time animation and rendering. To this end, we propose a compositional avatar representation, where the body, hands and the face are separately modeled in a way that the structural prior from parametric mesh templates is properly utilized without compromising representation flexibility. Furthermore, we disentangle the geometry and appearance for each part. With these technical designs, we propose a dedicated deferred rendering pipeline, which can be executed at a real-time framerate to synthesize high-quality free-view images. The disentanglement of geometry and appearance also allows us to design a two-pass training strategy that combines volume rendering and surface rendering for network training. In this way, patch-level supervision can be applied to force the network to learn sharp appearance details on the basis of geometry estimation. Overall, our method enables automatic construction of expressive full-body avatars with real-time rendering capability, and can generate photo-realistic images with dynamic details for novel body motions and facial expressions.
Funder
National Key R&D Program of China
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Graphics and Computer-Aided Design
Cited by
3 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. SFLSH: Shape-Dependent Soft-Flesh Avatars;SIGGRAPH Asia 2023 Conference Papers;2023-12-10
2. Recovering 3D Human Mesh From Monocular Images: A Survey;IEEE Transactions on Pattern Analysis and Machine Intelligence;2023-12
3. Effective Whole-body Pose Estimation with Two-stages Distillation;2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW);2023-10-02