Abstract
Understanding long videos requires extracting query-relevant information from long sequences under tight compute budgets. Existing text-then-LLM pipelines lose fine-grained visual cues, while video-based multimodal large language models (MLLMs) can keep visual details but are too frame-hungry and computationally expensive. In this work, we aim to harness MLLMs for efficient video understanding. We propose ProVCA, a progressive video condensation agent that iteratively locates key video frames at multiple granularities. ProVCA first adopts a segment localization module to identify the video segment relevant to the query, then a snippet selection module to select important snippets based on similarity, and finally a keyframe refinement module to pinpoint specific keyframes in those snippets. By progressively narrowing the scope from coarse segments to fine frames, ProVCA identifies a small set of keyframes for MLLM-based reasoning. ProVCA achieves state-of-the-art zero-shot accuracies of 69.3\% on EgoSchema, 80.5\% on NExT-QA, and 77.7\% on IntentQA, while using fewer frames than previous training-free methods.
Abstract (translated)
理解长视频需要在有限的计算预算下，从长序列中提取与查询相关的信息。现有的“文本处理+大语言模型”流程会丢失细粒度的视觉线索，而基于视频的多模态大语言模型（MLLMs）虽能保留视觉细节，却因帧数需求过高而计算成本昂贵。本研究旨在利用MLLMs实现高效视频理解。我们提出了ProVCA——一种渐进式视频压缩智能体，可迭代地在多种粒度下定位关键视频帧。ProVCA首先采用片段定位模块识别与查询相关的视频片段，接着通过片段选择模块基于相似度筛选重要片段，最终通过关键帧精炼模块在片段内确定具体关键帧。通过从粗粒度片段到细粒度帧逐步缩小范围，ProVCA为基于MLLM的推理识别出少量关键帧。在无需训练的条件下，ProVCA在EgoSchema上达到69.3%、NExT-QA上80.5%、IntentQA上77.7%的零样本准确率，且使用的帧数少于此前无训练方法。
URL
https://arxiv.org/abs/2604.02891
