Abstract
Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens. Existing pruning primary targets intra-frame spatial redundancy or prunes inside the LLM with shallow-layer overhead, yielding suboptimal spatiotemporal reduction and underutilizing long-context compressibility. All of them often discard subtle yet informative context from merged or pruned tokens. In this paper, we propose a new perspective that elaborates token \textbf{A}nchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global \textbf{O}ptimal \textbf{T}ransport (\textbf{AOT}). Specifically, we first establish local- and global-aware token anchors within each frame under the attention guidance, which then optimal transport aggregates the informative contexts from pruned tokens, constructing intra-frame token anchors. Then, building on the temporal frame clips, the first frame within each clip will be considered as the keyframe anchors to ensemble similar information from consecutive frames through optimal transport, while keeping distinct tokens to represent temporal dynamics, leading to efficient token reduction in a training-free manner. Extensive evaluations show that our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs, obtaining substantial computational efficiency while preserving temporal and visual fidelity. Project webpage: \href{this https URL}{AOT}.
Abstract (translated)
视频大型语言模型（VLLMs）展示了强大的视频理解能力，但由于冗余的视觉令牌导致效率低下。现有剪枝方法主要针对帧内空间冗余或在深度较浅的层进行剪枝，导致时空压缩效果不佳，并且未能充分利用长上下文可压缩性。它们经常从合并或裁剪的令牌中丢弃细微但有信息量的上下文。 本文提出了一种新的视角：通过局部-全局最优传输（AOT），详细阐述了帧内和帧间的令牌锚点，从而全面聚合信息上下文。具体而言，我们首先在注意力引导下，在每个视频帧内部建立具有局部和全局意识的令牌锚点，然后通过最优传输从被剪枝的令牌中汇总信息，构建帧内的令牌锚点。接着，在时间片段的基础上，将每个片段中的第一个帧视为关键帧锚点，利用最优传输整合连续帧中的相似信息，并保持区分性标记以表示时间动态变化，从而在无需训练的情况下实现有效的令牌减少。 广泛的评估表明，我们提出的AOT方法在各种短视频和长视频基准测试中，在领先的VLLMs上达到了具有竞争力的性能，同时获得了显著的计算效率并保留了时间和视觉保真度。项目网页：[此处应为实际链接，请访问项目页面了解更多信息](this https URL)。
URL
https://arxiv.org/abs/2603.01400
