Transcoding V-PCC Point Cloud Streams in Real-time

Abstract

Dynamic Point Clouds are a representation for 3D immersive media that allows users to freely navigate a scene while consuming the content. However, this comes at the cost of substantial data size, requiring efficient compression techniques to make point cloud videos accessible. Addressing this, Video-based Point Cloud Compression (V-PCC) projects points into 2D patches to compress video frames, leveraging the high compression efficiency of legacy video codecs and exploiting temporal correlations in the 2D images. However, clustering and projecting points into meaningful 2D patches is computationally intensive, leading to high encoding latency in V-PCC. Applying adaptive streaming techniques, originating from traditional video streaming, multiplies the computational effort as multiple encodings of the same content are required. In this light, transcoding a compressed representation into lower qualities for dynamic adaptation to user requirements is gaining popularity. To address the high latency when employing the full decoder-encoder stack of V-PCC during transcoding, we propose RABBIT, a novel technique that only re-encodes the underlying video sub-streams. This is in contrast to slow V-PCC transcoding that reconstructs and re-encodes the raw point cloud at a new quality setting. By eliminating expensive overhead resulting from calculations based on the 3D space representation, the latency of RABBIT is bounded by the latency of transcoding the underlying video streams, allowing optimized video codec implementations to be used to meet the real time requirements of adaptive streaming systems. Our evaluations of RABBIT, using various optimized video codec implementations, shows on-par quality with the baseline V-PCC transcoding given a high-quality representation. Given unicast or multicast distribution of a point cloud stream and in-network or edge transcoders, our evaluations show the tradeoff between rate-distortion performance and the required network bandwidth.