Affiliation:
1. Wangxuan Institute of Computer Technology, Peking University, Beijing, China
Abstract
Modern codecs remove temporal redundancy of a video via inter prediction,
i.e.,
searching previously coded frames for similar blocks and storing motion vectors to save bit-rates. However, existing codecs adopt block-level motion estimation, where a block is regressed by reference blocks linearly and is doomed to fail to deal with non-linear motions. In this article, we generate virtual reference frames (VRFs) with previously reconstructed frames via deep networks to offer an additional candidate, which is not constrained to linear motion structure and further significantly improves coding efficiency. More specifically, we propose a novel deep Auto-Regressive Moving-Average (ARMA) model, Error-Corrected Auto-Regressive Network (ECAR-Net), equipped with the powers of the conventional statistic ARMA models and deep networks jointly for reference frame prediction. Similar to conventional ARMA models, the ECAR-Net consists of two stages: Auto-Regression (AR) stage and Error-Correction (EC) stage, where the first part predicts the signal at the current time-step based on previously reconstructed frames, while the second one compensates for the output of the AR stage to obtain finer details. Different from the statistic AR models only focusing on short-term temporal dependency, the AR model of our ECAR-Net is further injected with the long-term dynamics mechanism, where long temporal information is utilized to help predict motions more accurately. Furthermore, ECAR-Net works in a configuration-adaptive way,
i.e.,
using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios. With the well-designed network, our method surpasses HEVC on average 5.0% and 6.6% BD-rate saving for the luma component under the Low Delay B and Random Access configurations and also obtains on average 1.54% BD-rate saving over VVC. Furthermore, ECAR-Net works in a configuration-adaptive way,
i.e.,
using different dynamics and error definitions for the Low Delay B and Random Access configurations, which helps improve the adaptivity and generality in diverse coding scenarios.
Funder
National Key Research and Development Program of China
National Natural Science Foundation of China
Research achievement of Key Laboratory of Science, Techonology and Standard in Press Industry
State Key Laboratory of Media Convergence Production Technology and Systems
Publisher
Association for Computing Machinery (ACM)
Subject
Computer Networks and Communications,Hardware and Architecture
Reference49 articles.
1. Overview of the H.264/AVC video coding standard
2. Overview of the High Efficiency Video Coding (HEVC) Standard
3. Benjamin Bross, Jianle Chen, Shan Liu, and Ye-Kui Wang. 2020. Versatile video coding (draft 9). In Proceedings of the Document JVET-R2001.
4. X. Zhang, W. Yang, Y. Hu, and J. Liu. 2018. Dmcnn: Dual-domain multi-scale convolutional neural network for compression artifacts removal. In Proceedings of the IEEE International Conference on Image Processing. IEEE, 390–394.
5. Image/video restoration via multiplanar autoregressive model and low-rank optimization;Li Mading;ACM Transactions on Multimedia Computing, Communications, and Applications,2019
Cited by
6 articles.
订阅此论文施引文献
订阅此论文施引文献,注册后可以免费订阅5篇论文的施引文献,订阅后可以查看论文全部施引文献
1. Deep video compression based on Long-range Temporal Context Learning;Computer Vision and Image Understanding;2024-11
2. Towards Deep Reference Frame in Versatile Video Coding NNVC;2023 IEEE International Conference on Visual Communications and Image Processing (VCIP);2023-12-04
3. Deep Reference Frame Generation Method for VVC Inter Prediction Enhancement;IEEE Transactions on Circuits and Systems for Video Technology;2023
4. Deep Reference Frame Interpolation based Inter Prediction Enhancement for Versatile Video Coding;2022 IEEE International Conference on Visual Communications and Image Processing (VCIP);2022-12-13
5. Context-aware Pseudo-true Video Interpolation at 6G Edge;ACM Transactions on Multimedia Computing, Communications, and Applications;2022-10-31