Bridging Formal Shape Models and Deep Learning: A Novel Fusion for Understanding 3D Objects
Author:
Zhang Jincheng1, Willis Andrew R.1ORCID
Affiliation:
1. Department of Electrical and Computer Engineering, University of North Carolina at Charlotte, Charlotte, NC 28223, USA
Abstract
This article describes a novel fusion of a generative formal model for three-dimensional (3D) shapes with deep learning (DL) methods to understand the geometric structure of 3D objects and the relationships between their components, given a collection of unorganized point cloud measurements. Formal 3D shape models are implemented as shape grammar programs written in Procedural Shape Modeling Language (PSML). Users write PSML programs to describe complex objects, and DL networks estimate the configured free parameters of the program to generate 3D shapes. Users write PSML programs to enforce fundamental rules that define an object class and encode object attributes, including shapes, components, size, position, etc., into a parametric representation of objects. This fusion of the generative model with DL offers artificial intelligence (AI) models an opportunity to better understand the geometric organization of objects in terms of their components and their relationships to other objects. This approach allows human-in-the-loop control over DL estimates by specifying lists of candidate objects, the shape variations that each object can exhibit, and the level of detail or, equivalently, dimension of the latent representation of the shape. The results demonstrate the advantages of the proposed method over competing approaches.
Reference45 articles.
1. Ajayi, E.A., Lim, K.M., Chong, S.C., and Lee, C.P. (2023). 3D Shape Generation via Variational Autoencoder with Signed Distance Function Relativistic Average Generative Adversarial Network. Appl. Sci., 13. 2. Dai, B., and Wipf, D. (2019). Diagnosing and enhancing VAE models. arXiv. 3. Kosiorek, A.R., Strathmann, H., Zoran, D., Moreno, P., Schneider, R., Mokrá, S., and Rezende, D.J. (2021, January 18–24). Nerf-vae: A geometry aware 3d scene generative model. Proceedings of the International Conference on Machine Learning, Virtual. 4. Wu, J., Zhang, C., Xue, T., Freeman, B., and Tenenbaum, J. (2016). Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling. Adv. Neural Inf. Process. Syst., 29. 5. Frühstück, A., Sarafianos, N., Xu, Y., Wonka, P., and Tung, T. (2023, January 17–24). Vive3d: Viewpoint-independent video editing using 3d-aware gans. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, BC, Canada.
|
|