Physically Based Rendering of Animated Point Clouds for EXtended Reality

Abstract

Abstract Point cloud 3D models are gaining increasing popularity due to the proliferation of scanning systems in various fields, including autonomous vehicles and robotics. When employed for rendering purposes, point clouds are typically depicted with their original colors acquired during the acquisition, often without taking into account the lighting conditions of the scene in which the model is situated. This can result in a lack of realism in numerous contexts, especially when dealing with animated point clouds used in eXtended reality applications, where it is desirable for the model to respond to incoming light and seamlessly blend with the surrounding environment. This paper proposes the application of physically based rendering (PBR), a rendering technique widely used in real-time computer graphics applications, to animated point cloud models for reproducing specular reflections, and achieving a photo-realistic and physically accurate look under any lighting condition. To achieve this, we first explore the extension of commonly used animated point cloud formats to incorporate normal vectors and PBR parameters, like roughness and metalness. Additionally, the encoding of the animated environment maps necessary for the PBR technique is investigated. Then, an animated point cloud model is rendered with a shader implementing the proposed PBR method. Finally, we compare the outcomes of this PBR pipeline with traditional renderings of the same point cloud produced using commonly used shaders, taking into account different lighting conditions and environments. Through these comparisons, we demonstrate how the proposed PBR method enhances the visual integration of the point cloud with its surroundings. Furthermore, it will be shown that using this rendering technique, it is possible to render different materials, by exploiting the features of PBR and the encoding of the surrounding environment.