Abstract
Implicit Neural Networks (INRs) have emerged as powerful representations to encode all forms of data, including images, videos, audios, and scenes. With video, many INRs for video have been proposed for the compression task, and recent methods feature significant improvements with respect to encoding time, storage, and reconstruction quality. However, these encoded representations lack semantic meaning, so they cannot be used for any downstream tasks that require such properties, such as retrieval. This can act as a barrier for adoption of video INRs over traditional codecs as they do not offer any significant edge apart from compression. To alleviate this, we propose a flexible framework that decouples the spatial and temporal aspects of the video INR. We accomplish this with a dictionary of per-frame latents that are learned jointly with a set of video specific hypernetworks, such that given a latent, these hypernetworks can predict the INR weights to reconstruct the given frame. This framework not only retains the compression efficiency, but the learned latents can be aligned with features from large vision models, which grants them discriminative properties. We align these latents with CLIP and show good performance for both compression and video retrieval tasks. By aligning with VideoLlama, we are able to perform open-ended chat with our learned latents as the visual inputs. Additionally, the learned latents serve as a proxy for the underlying weights, allowing us perform tasks like video interpolation. These semantic properties and applications, existing simultaneously with ability to perform compression, interpolation, and superresolution properties, are a first in this field of work.
Abstract (translated)
隐式神经网络（INRs）已成为对所有数据形式进行编码的有力量的表示，包括图像、视频、音频和场景。对于视频，许多INRs在压缩任务中已经提出，并且最近的方法相对于编码时间、存储和重构质量有显著的改进。然而，这些编码表示缺乏语义意义，因此它们无法用于任何需要这种特性的下游任务，例如检索。这可能会成为采用视频INRs的传统编码的障碍，因为它们除了压缩作用外没有提供任何显著的优点。为了减轻这种障碍，我们提出了一个灵活的框架，将视频INR的空间和时间方面解耦。我们通过学习与视频特定超网络共同学习的每个帧的潜在词典来完成这一目标，使得给定潜在，这些超网络可以预测INR权重以重构给定的帧。这个框架不仅保留了压缩效率，而且学习到的潜在可以与大型视觉模型的特征对齐，从而赋予它们判别性质。我们将这些潜在与CLIP对齐，并展示良好的压缩和视频检索任务性能。通过与VideoLlama对齐，我们能够与学习到的潜在进行 open-ended chat，作为视觉输入。此外，学习到的潜在还作为底层重量的代理，使我们能够执行视频平滑等任务。这些语义特性和应用，与压缩能力、平滑和超分辨率特性同时出现，是这一领域的工作中的第一次。
URL
https://arxiv.org/abs/2408.02672
