3D SCENE RECONSTRUCTION AND DIGITIZATION METHOD FOR MIXED REALITY SYSTEMS

Abstract

Mixed reality systems are a promising direction of research that opens up great opportunities forinteraction with virtual objects in the real world. Like any promising direction, mixed reality has a number of unresolved problems. One of these problems is the synthesis of natural lighting conditions for virtual objects, including the correct light interaction of virtual objects with the real world. Since virtual and real objects are located in different spaces, it is difficult to ensure their correct interaction. To create digital copies of realworld objects, machine learning tools and neural network technologies are employed. These methods are successfully used in computer vision for space orientation and environment reconstruction. As a solution, it is proposed to transfer all objects into the same information space: virtual space. This makes it possible to solve most of the problems associated with visual discomfort caused by the unnatural light interaction of real and virtual objects. Thus, the basic idea of the method is to recognize physical objects from point clouds and replace these objects with virtual CAD models. In other words, it implies semantic analysis of a scene and classification of objects with their subsequent transformation into polygonal models. In this study, we use competitive neural network architectures, which can provide state-of-the-art results. The test experiments are carried out on Semantic3D, ScanNet, and S3DIS, which are currently the largest datasets with point clouds that represent indoor scenes. For semantic segmentation and classification of 3D point clouds, we use the PointNeXt architecture based on PointNet, as well as modern methods of data augmentation in the process of learning. For geometry reconstruction, the Soft Rasterizer differentiable rendering method and the Total3Understanding neural network are considered.