Abstract
We present a neural scene graph---a modular and controllable representation of scenes with elements that are learned from data. We focus on the forward rendering problem, where the scene graph is provided by the user and references learned elements. The elements correspond to geometry and material definitions of scene objects and constitute the leaves of the graph; we store them as high-dimensional vectors. The position and appearance of scene objects can be adjusted in an artist-friendly manner via familiar transformations, e.g. translation, bending, or color hue shift, which are stored in the inner nodes of the graph. In order to apply a (non-linear) transformation to a learned vector, we adopt the concept of linearizing a problem by lifting it into higher dimensions: we first encode the transformation into a high-dimensional matrix and then apply it by standard matrix-vector multiplication. The transformations are encoded using neural networks. We render the scene graph using a streaming neural renderer, which can handle graphs with a varying number of objects, and thereby facilitates scalability. Our results demonstrate a precise control over the learned object representations in a number of animated 2D and 3D scenes. Despite the limited visual complexity, our work presents a step towards marrying traditional editing mechanisms with learned representations, and towards high-quality, controllable neural rendering.
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Graphics and Computer-Aided Design
Cited by
7 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. LightFormer: Light-Oriented Global Neural Rendering in Dynamic Scene;ACM Transactions on Graphics;2024-07-19
2. SGRec3D: Self-Supervised 3D Scene Graph Learning via Object-Level Scene Reconstruction;2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV);2024-01-03
3. NeLT: Object-Oriented Neural Light Transfer;ACM Transactions on Graphics;2023-08-29
4. Deep Appearance Prefiltering;ACM Transactions on Graphics;2023-01-16
5. Novel View Synthesis of Human Interactions from Sparse Multi-view Videos;Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings;2022-08-07