Foveated Path Culling: A mixed path tracing and radiance field approach for optimizing rendering in XR Displays

Abstract

Real-time effects achieved by path tracing are essential for creating highly accurate illumination effects in interactive environments. However, due to its computational complexity, it is essential to explore optimization techniques like Foveated Rendering when considering Head Mounted Displays. In this paper we combine traditional Foveated Rendering approaches with recent advancements in the field of radiance fields, extending a previous work and including recent advancements based on Gaussian Splatting. The present paper proposes the usage of mixing real time path tracing at the fovea region of an HMD while replacing the images at the peripheral by pre-computed radiance fields, inferred by neural networks or rendered in real time due to Gaussian splats. We name our approach as Foveated Path Culling (FPC) due to the process of culling raycasts, diminishing the workload by replacing most of the screen raytracing tasks by a less costly approach. FPC allowed us for better frame rates when compared to purely path tracing while rendering scenes in real time, increasing the frame rate speedup proportionally to the display resolution. Our work contributes to the development of rendering techniques for XR experiences that demand low latency, high resolution and high visual quality through global illumination effects.