Abstract
With recent advancements in video backbone architectures, combined with the remarkable achievements of large language models (LLMs), the analysis of long-form videos spanning tens of minutes has become both feasible and increasingly prevalent. However, the inherently redundant nature of video sequences poses significant challenges for contemporary state-of-the-art models. These challenges stem from two primary aspects: 1) efficiently incorporating a larger number of frames within memory constraints, and 2) extracting discriminative information from the vast volume of input data. In this paper, we introduce a novel end-to-end schema for long-form video understanding, which includes an information-density-based adaptive video sampler (AVS) and an autoencoder-based spatiotemporal video compressor (SVC) integrated with a multimodal large language model (MLLM). Our proposed system offers two major advantages: it adaptively and effectively captures essential information from video sequences of varying durations, and it achieves high compression rates while preserving crucial discriminative information. The proposed framework demonstrates promising performance across various benchmarks, excelling in both long-form video understanding tasks and standard video understanding benchmarks. These results underscore the versatility and efficacy of our approach, particularly in managing the complexities of prolonged video sequences.
Abstract (translated)
随着视频骨干网络架构的最新进展以及大型语言模型（LLM）的显著成就，对长达数分钟的长视频进行分析变得既可行又日益普遍。然而，视频序列本身固有的冗余性为现代最先进的模型带来了重大挑战。这些挑战主要源自两个方面：1) 在内存限制下有效整合更多帧，2) 从大量的输入数据中提取判别信息。 在本文中，我们提出了一种新的端到端长视频理解方案，包括基于信息密度的自适应视频采样器（AVS）和与多模态大型语言模型（MLLM）集成的空间时间视频压缩器（SVC）。我们的系统具有两个主要优势：它能够根据不同长度的视频序列自适应且有效地捕捉重要信息，并在保持关键判别性信息的同时实现高压缩率。所提出的框架在各种基准测试中表现出色，尤其擅长长视频理解任务和标准视频理解基准。 这些结果强调了我们方法的灵活性和有效性，特别是在处理长期视频序列复杂性的方面。
URL
https://arxiv.org/abs/2602.17869
