Abstract
Video Large Language Models (VLMs) have achieved remarkable success in video understanding, but the significant computational cost from processing dense frames severely limits their practical application. Existing methods alleviate this by selecting keyframes, but their greedy decision-making, combined with a decoupled evaluation of relevance and diversity, often falls into local optima and results in erroneously selecting irrelevant noise frames. To address these challenges, we propose GIFT: Global Irreplaceability Frame Targeting, a novel training-free framework that selects frames by assessing their intrinsic irreplaceability. Specifically, we first introduce Directed Diversity to quantify a frame's uniqueness conditioned on relevance, which allows us to formulate a unified irreplaceability score. Subsequently, our Budget-Aware Refinement strategy employs a adaptive iterative process that first secures a core set of frames with the highest irreplaceability, and then shifts its priority to building crucial temporal context around these selections as the budget expands. Extensive experiments demonstrate that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.
Abstract (translated)
视频大语言模型（VLMs）在视频理解领域已取得显著成功，但处理密集帧所带来的巨大计算成本严重制约了其实际应用。现有方法通过选择关键帧来缓解此问题，然而其贪婪决策结合对相关性和多样性的解耦评估，常陷入局部最优并导致错误选择无关噪声帧。为应对这些挑战，我们提出GIFT：全局不可替代帧定位，一种新颖的无训练框架，通过评估帧的内在不可替代性进行选择。具体而言，我们首先引入导向多样性（Directed Diversity）来量化在相关性条件下的帧独特性，从而构建统一的不可替代性评分。随后，我们的预算感知优化策略（Budget-Aware Refinement）采用自适应迭代过程：首先锁定不可替代性最高的核心帧集，随后随预算扩展将优先级转向围绕这些选择构建关键时序上下文。大量实验表明，在长视频基准测试中，GIFT在LLaVA-Video-7B上相比均匀采样实现了最高12.5%的平均提升。
URL
https://arxiv.org/abs/2603.25072
