VT‐NeRF: Neural radiance field with a vertex‐texture latent code for high‐fidelity dynamic human‐body rendering

Abstract

AbstractThe fusion of a human prior with neural rendering techniques has recently emerged as one of the most promising approaches to processing dynamic human‐body scenes with sparse inputs. However, learning geometric details and appearance in dynamic human‐body scenes based solely on a human prior model represents a severely under‐constrained problem. A new human‐body representation method to solve this problem: a neural radiance field with vertex‐texture latent codes (VT‐NeRF) is proposed. VT‐NeRF uses joint latent code to improve access to detailed information, combining vertex latent codes with 2D texture latent codes for the body surface. Referencing a 3D human skeleton for accurate guidance, the human model can quickly match poses and learn information about the body in different frames. VT‐NeRF can integrate body information from different frames and different poses quickly because it uses an information‐rich human prior: a 3D human skeleton and parametric models. A 3D human scene is then presented as an implied field of density and colour. Experiments with the ZJU‐MoCap dataset show that our method outperforms previous methods in terms of both novel‐view synthesis and 3D human reconstruction quality. It is twice as fast as Neural Body, and its average accuracy reaches 95.9%.