Abstract
Frame selection is crucial due to high frame redundancy and limited context windows when applying Large Vision-Language Models (LVLMs) to long videos. Current methods typically select frames with high relevance to a given query, resulting in a disjointed set of frames that disregard the narrative structure of video. In this paper, we introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework that presents a new perspective: effective video understanding hinges not only on high relevance but, more importantly, on capturing semantic shifts - pivotal moments of narrative change that are essential to comprehending the holistic storyline of video. However, direct detection of abrupt changes in the query-frame similarity signal is often unreliable due to high-frequency noise arising from model uncertainty and transient visual variations. To address this, we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains. By applying this transform, we decompose the noisy signal into multiple scales and extract a clean semantic change signal from the coarsest scale. We identify the local extrema of this signal as semantic boundaries, which segment the video into coherent clips. Building on this, WFS-SB comprises a two-stage strategy: first, adaptively allocating a frame budget to each clip based on a composite importance score; and second, within each clip, employing the Maximal Marginal Relevance approach to select a diverse yet relevant set of frames. Extensive experiments show that WFS-SB significantly boosts LVLM performance, e.g., improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench, consistently outperforming state-of-the-art methods.
Abstract (translated)
帧选择在应用大型视觉-语言模型（LVLM）到长视频时至关重要，因为这可以减少冗余的帧数并适应有限的上下文窗口。当前的方法通常会选择与给定查询高度相关的帧，导致所选帧集合缺乏连贯性，忽略了视频叙事结构的重要性。在这篇论文中，我们提出了基于小波变换检测语义边界（WFS-SB）的训练自由框架，它提供了一个新的视角：有效的视频理解不仅依赖于高相关性，更重要的是捕捉到叙述变化的关键时刻——这些关键时刻对于理解视频的整体故事情节至关重要。 然而，直接从查询-帧相似度信号中检测突变通常会因为模型不确定性以及短暂视觉变化产生的高频噪声而变得不可靠。为了应对这一挑战，我们利用了小波变换，这种方法通过在时间和频率域的多分辨率分析提供了理想解决方案。通过对这种变换的应用，我们将嘈杂信号分解为多个尺度，并从最粗糙尺度中提取出干净的语义变化信号。接着，我们识别该信号中的局部极值作为语义边界，这些边界将视频分割成连贯的片段。 在此基础上，WFS-SB采用了两阶段策略：首先，在每个片段内根据综合重要性分数自适应地分配帧预算；其次，在每一个片段内部应用最大边缘相关方法来选择多样且相关的帧集合。广泛的实验结果表明，WFS-SB显著提升了LVLM的表现力，例如在VideoMME上提高了5.5%的准确性、在MLVU上提高了9.5%，以及在LongVideoBench上提高了6.2%，并且持续优于现有的最佳方法。 该框架通过更加关注视频中叙述结构的变化点，实现了更高效且连贯的帧选择，从而为LVLM处理长视频提供了更为有效的解决方案。
URL
https://arxiv.org/abs/2603.00512
