Abstract
Implicit neural representations for video (NeRV) have recently become a novel way for high-quality video representation. However, existing works employ a single network to represent the entire video, which implicitly confuse static and dynamic information. This leads to an inability to effectively compress the redundant static information and lack the explicitly modeling of global temporal-coherent dynamic details. To solve above problems, we propose DS-NeRV, which decomposes videos into sparse learnable static codes and dynamic codes without the need for explicit optical flow or residual supervision. By setting different sampling rates for two codes and applying weighted sum and interpolation sampling methods, DS-NeRV efficiently utilizes redundant static information while maintaining high-frequency details. Additionally, we design a cross-channel attention-based (CCA) fusion module to efficiently fuse these two codes for frame decoding. Our approach achieves a high quality reconstruction of 31.2 PSNR with only 0.35M parameters thanks to separate static and dynamic codes representation and outperforms existing NeRV methods in many downstream tasks. Our project website is at this https URL.
Abstract (translated)
Implicit neural representations for video (NeRV) have recently become a novel way for high-quality video representation. However, existing works employ a single network to represent the entire video, which implicitly confuses static and dynamic information. This leads to an inability to effectively compress the redundant static information and lack the explicitly modeling of global temporal-coherent dynamic details. 为了解决上述问题，我们提出了DS-NeRV，它将视频分解为稀疏可学习静态代码和动态代码，无需显式光流或残差监督。通过设置两个代码的不同采样率，并应用加权求和插值采样方法，DS-NeRV有效地利用冗余静态信息，同时保留高频细节。此外，我们还设计了一个跨通道关注（CCA）融合模块，用于有效地融合这两个代码进行帧解码。 我们的方法通过单独的静态和动态代码表示实现了31.2 PSNR的高质量重建，同时仅使用0.35M个参数。这使得我们在许多下游任务中超过了现有的NeRV方法。我们的项目网站是https://www.google.com/url?q=https://github.com/dantianzeng/DS-NeRV。
URL
https://arxiv.org/abs/2403.15679
