Abstract
Training-free video understanding leverages the strong image comprehension capabilities of pre-trained vision language models (VLMs) by treating a video as a sequence of static frames, thus obviating the need for costly video-specific training. However, this paradigm often suffers from severe visual redundancy and high computational overhead, especially when processing long videos. Crucially, existing keyframe selection strategies, especially those based on CLIP similarity, are prone to biases and may inadvertently overlook critical frames, resulting in suboptimal video comprehension. To address these significant challenges, we propose \textbf{KTV}, a novel two-stage framework for efficient and effective training-free video understanding. In the first stage, KTV performs question-agnostic keyframe selection by clustering frame-level visual features, yielding a compact, diverse, and representative subset of frames that mitigates temporal redundancy. In the second stage, KTV applies key visual token selection, pruning redundant or less informative tokens from each selected keyframe based on token importance and redundancy, which significantly reduces the number of tokens fed into the LLM. Extensive experiments on the Multiple-Choice VideoQA task demonstrate that KTV outperforms state-of-the-art training-free baselines while using significantly fewer visual tokens, \emph{e.g.}, only 504 visual tokens for a 60-min video with 10800 frames, achieving $44.8\%$ accuracy on the MLVU-Test benchmark. In particular, KTV also exceeds several training-based approaches on certain benchmarks.
Abstract (translated)
无训练视频理解方法利用预训练的视觉语言模型（VLM）的强大图像理解能力，将视频视为一系列静态帧序列，从而避免了昂贵的特定于视频的训练需求。然而，这种范式在处理长视频时往往遭受严重的视觉冗余和高昂的计算开销问题。特别是基于CLIP相似性的现有关键帧选择策略容易产生偏差，并可能无意中忽略关键帧，导致次优的视频理解结果。 为了解决这些重大挑战，我们提出了一种名为KTV的新颖两阶段框架，用于高效的无训练视频理解。在第一阶段，KTV通过聚类帧级别的视觉特征执行与问题无关的关键帧选择，从而产生一个精简、多样且具代表性的关键帧子集，有效减少了时间冗余。在第二阶段，KTV应用关键视觉标记选择方法，根据每个选定关键帧中令牌的重要性及冗余度去除多余的或信息量较小的令牌，显著减少输入LLM（大型语言模型）中的视觉令牌数量。 广泛的实验显示，在多项选择视频问答任务上，KTV的表现超越了最先进的无训练基准，并且使用了显著较少的视觉令牌。例如，对于一个包含10800帧的60分钟视频，仅需504个视觉令牌即可在MLVU-Test基准测试中达到44.8%的准确率。此外，在某些基准上，KTV的表现甚至超过了几个基于训练的方法。 这个描述突出了KTV框架如何通过两个阶段有效减少冗余，并提高无训练视频理解模型的效果和效率。
URL
https://arxiv.org/abs/2602.03615
