Abstract
Diffusion models have achieved remarkable success in generating high quality image and video data. More recently, they have also been used for image compression with high perceptual quality. In this paper, we present a novel approach to extreme video compression leveraging the predictive power of diffusion-based generative models at the decoder. The conditional diffusion model takes several neural compressed frames and generates subsequent frames. When the reconstruction quality drops below the desired level, new frames are encoded to restart prediction. The entire video is sequentially encoded to achieve a visually pleasing reconstruction, considering perceptual quality metrics such as the learned perceptual image patch similarity (LPIPS) and the Frechet video distance (FVD), at bit rates as low as 0.02 bits per pixel (bpp). Experimental results demonstrate the effectiveness of the proposed scheme compared to standard codecs such as H.264 and H.265 in the low bpp regime. The results showcase the potential of exploiting the temporal relations in video data using generative models. Code is available at: this https URL
Abstract (translated)
扩散模型在生成高质量图像和视频数据方面取得了显著的成功。更最近，它们还被用于具有高感知质量的图像压缩。在本文中，我们提出了一种利用扩散基于生成模型的预测能力来实现极端视频压缩的新方法。条件扩散模型对几个神经压缩帧进行编码，生成后续帧。当重建质量低于期望水平时，新帧被编码以重新启动预测。整个视频按位率序列编码以实现视觉上令人愉悦的重建，考虑学习到的感知图像补丁相似度（LPIPS）和费希特视频距离（FVD）等感知质量指标， bit rates在0.02 bit/pixel（bpp）时。实验结果表明，与低bpp范围内的标准编解码器（如H.264和H.265）相比，所提出的方案在bpp低端具有有效的效果。结果突出了在视频数据中利用生成模型的时间关系潜力。代码可在此处下载：https:// this URL
URL
https://arxiv.org/abs/2402.08934
