Abstract
Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework grounded in information bottleneck theory. We formulate keyframe selection as maximizing the conditional mutual information between selected frames and the query, providing a principled objective that reflects each frame's contribution to answering the question. To make this objective tractable, we exploit its structure to derive a decomposed optimization that reduces subset selection to independent frame-level scoring. We further introduce a query-conditioned evidence scoring network trained with a contrastive objective to estimate evidential importance efficiently. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under strict token budgets, while significantly improving training efficiency.
Abstract (translated)
多模态大语言模型（MLLMs）在视频问答任务中展现出卓越性能，但其应用于长视频时受限于上下文长度与计算成本，使得关键帧采样成为必要环节。现有方法通常依赖语义相关性或强化学习，前者难以捕捉证据性线索，后者则面临组合优化效率低下的问题。本研究提出一个基于信息瓶颈理论的证据驱动关键帧采样框架，将关键帧选择建模为最大化所选帧与查询之间的条件互信息，从而建立能反映各帧对回答问题贡献的原则性目标。为实现该目标的可计算性，我们利用其结构特性推导出分解优化方案，将子集选择问题转化为独立的帧级评分。进一步引入经对比学习目标训练的查询条件证据评分网络，以高效估计帧的证据重要性。在长视频理解基准测试中，本方法在严格令牌预算下持续优于现有采样策略，同时显著提升训练效率。
URL
https://arxiv.org/abs/2604.01002
