Abstract
Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.
Abstract (translated)
大型视觉-语言模型（VLMs）正日益应用于长视频问答任务，但其推理过程常因输入帧数及由此产生的视觉标记数量而受限。简单的稀疏采样可能遗漏关键片段，而纯粹基于相关性的选择则容易陷入近重复帧，并牺牲时间上相距较远的证据覆盖。我们提出一种问题自适应的贪心帧选择方法，在固定帧数预算下联合优化查询相关性与语义代表性。该方法构建一个精确对齐时间戳的1FPS候选池（上限1000帧），将候选帧嵌入两个互补空间（SigLIP用于问题相关性，DINOv2用于语义相似性），并通过贪心最大化一个加权和来选择帧，该加权和由模块化相关性项与设施位置覆盖项组成。此目标函数具有归一化、单调和子模特性，因此享有标准(1-1/e)贪心近似保证。为适应不同问题对相关性与覆盖的权衡需求，我们引入四种预设策略及一个轻量级纯文本问题类型分类器，将每个查询路由至其表现最佳的预设策略。在MLVU上的实验表明，相比均匀采样和近期强基线，我们的方法在各类帧数预算下均持续提升准确率，且在预算紧张时改进最为显著。
URL
https://arxiv.org/abs/2603.20180
