Abstract
Long video understanding remains challenging for multimodal large language models (MLLMs) due to limited context windows, which necessitate identifying sparse query-relevant video segments. However, existing methods predominantly localize clues based solely on the query, overlooking the video's intrinsic structure and varying relevance across segments. To address this, we propose VideoDetective, a framework that integrates query-to-segment relevance and inter-segment affinity for effective clue hunting in long-video question answering. Specifically, we divide a video into various segments and represent them as a visual-temporal affinity graph built from visual similarity and temporal proximity. We then perform a Hypothesis-Verification-Refinement loop to estimate relevance scores of observed segments to the query and propagate them to unseen segments, yielding a global relevance distribution that guides the localization of the most critical segments for final answering with sparse observation. Experiments show our method consistently achieves substantial gains across a wide range of mainstream MLLMs on representative benchmarks, with accuracy improvements of up to 7.5% on VideoMME-long. Our code is available at this https URL
Abstract (translated)
长视频理解对于多模态大语言模型（MLLMs）而言仍是一项挑战，这主要受限于上下文窗口，使得模型必须识别出与查询稀疏相关的视频片段。然而，现有方法大多仅基于查询定位线索，忽视了视频的内在结构以及不同片段间相关性的差异。为解决这一问题，我们提出了VideoDetective框架，该框架整合了查询与片段的相关性以及片段间的关联性，从而在长视频问答中实现高效线索检索。具体而言，我们将视频划分为多个片段，并将其表示为基于视觉相似性与时序邻近性构建的视觉-时序关联图。随后，我们通过假设-验证-优化循环来估算已观测片段与查询的相关性分数，并将这些分数传播至未观测片段，从而获得全局相关性分布，以指导在稀疏观测条件下定位对最终回答最关键的片段。实验表明，我们的方法在多个主流MLLMs和代表性基准测试上均取得了显著提升，在VideoMME-long数据集上的准确率最高提升达7.5%。代码已公开于此链接。
URL
https://arxiv.org/abs/2603.22285
