Affiliation:
1. Max Planck Institute for Informatics
2. Stanford University, CA, USA
Abstract
We present the first real-time human performance capture approach that reconstructs dense, space-time coherent deforming geometry of entire humans in general everyday clothing from just a single RGB video. We propose a novel two-stage analysis-by-synthesis optimization whose formulation and implementation are designed for high performance. In the first stage, a skinned template model is jointly fitted to background subtracted input video, 2D and 3D skeleton joint positions found using a deep neural network, and a set of sparse facial landmark detections. In the second stage, dense non-rigid 3D deformations of skin and even loose apparel are captured based on a novel real-time capable algorithm for non-rigid tracking using dense photometric and silhouette constraints. Our novel energy formulation leverages automatically identified material regions on the template to model the differing non-rigid deformation behavior of skin and apparel. The two resulting non-linear optimization problems per frame are solved with specially tailored data-parallel Gauss-Newton solvers. To achieve real-time performance of over 25Hz, we design a pipelined parallel architecture using the CPU and two commodity GPUs. Our method is the first real-time monocular approach for full-body performance capture. Our method yields comparable accuracy with off-line performance capture techniques while being orders of magnitude faster.
Funder
European Research Council
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Graphics and Computer-Aided Design
Cited by
145 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. LayerNet: High-Resolution Semantic 3D Reconstruction of Clothed People;IEEE Transactions on Pattern Analysis and Machine Intelligence;2024-02
2. SAILOR: Synergizing Radiance and Occupancy Fields for Live Human Performance Capture;ACM Transactions on Graphics;2023-12-05
3. Fusing exocentric and egocentric real-time reconstructions for embodied immersive experiences;2023 38th International Conference on Image and Vision Computing New Zealand (IVCNZ);2023-11-29
4. MVP-Human Dataset for 3-D Clothed Human Avatar Reconstruction From Multiple Frames;IEEE Transactions on Biometrics, Behavior, and Identity Science;2023-10
5. HDHumans;Proceedings of the ACM on Computer Graphics and Interactive Techniques;2023-08-16