Abstract
Long-form video understanding is essential for various applications such as video retrieval, summarizing, and question answering. Yet, traditional approaches demand substantial computing power and are often bottlenecked by GPU memory. To tackle this issue, we present Long-Video Memory Network, Long-VMNet, a novel video understanding method that employs a fixed-size memory representation to store discriminative patches sampled from the input video. Long-VMNet achieves improved efficiency by leveraging a neural sampler that identifies discriminative tokens. Additionally, Long-VMNet only needs one scan through the video, greatly boosting efficiency. Our results on the Rest-ADL dataset demonstrate an 18x -- 75x improvement in inference times for long-form video retrieval and answering questions, with a competitive predictive performance.
Abstract (translated)
长视频理解对于视频检索、摘要生成和问答等应用至关重要。然而，传统方法需要大量的计算资源，并且常常受到GPU内存的限制。为了解决这一问题，我们提出了一种名为Long-Video Memory Network（简称Long-VMNet）的新颖视频理解方法。该方法采用固定大小的记忆表示来存储从输入视频中采样的具有区分性的补丁(patch)。通过利用一个能够识别具有区分性令牌(token)的神经采样器，Long-VMNet实现了更高的效率。此外，Long-VMNet只需对视频进行一次扫描，极大地提高了效率。在Rest-ADL数据集上的实验结果表明，在长视频检索和问答任务上，Long-VMNet相比传统方法提升了18到75倍的推理速度，并且预测性能仍然保持竞争力。
URL
https://arxiv.org/abs/2503.13707
