Neutral Facial Rigging from Limited Spatiotemporal Meshes

Abstract

Manual facial rigging is time-consuming. Traditional automatic rigging methods lack either 3D datasets or explainable semantic parameters, which makes it difficult to retarget a certain 3D expression to a new face. To address the problem, we automatically generate a large 3D dataset containing semantic parameters, joint positions, and vertex positions from a limited number of spatiotemporal meshes. We establish an expression generator based on a multilayer perceptron with vertex constraints from the semantic parameters to the joint positions and establish an expression recognizer based on a generative adversarial structure from the joint positions to the semantic parameters. To enhance the accuracy of key facial area recognition, we add local vertex constraints for the eyes and lips, which are determined by the 3D masks computed by the proposed projection-searching algorithm. We testthe generation and recognition effects on a limited number of publicly available Metahuman meshes and self-collected meshes. Compared with existing methods, our generator has the shortest generation time of 14.78 ms and the smallest vertex relative mean square error of 1.57 × 10−3, while our recognizer has the highest accuracy of 92.92%. The ablation experiment verifies that the local constraints can improve the recognition accuracy by 3.02%. Compared with other 3D mask selection methods, the recognition accuracy is improved by 1.03%. In addition, our method shows robust results for meshes of different levels of detail, and the rig has more dimensions of semantic space. The source code will be made available if this paper is accepted for publication.