Abstract
In long-video understanding, conventional uniform frame sampling often fails to capture key visual evidence, leading to degraded performance and increased hallucinations. To address this, recent agentic thinking-with-videos paradigms have emerged, adopting a localize-clip-answer pipeline in which the model actively identifies relevant video segments, performs dense sampling within those clips, and then produces answers. However, existing methods remain inefficient, suffer from weak localization, and adhere to rigid workflows. To solve these issues, we propose VideoTemp-o3, a unified agentic thinking-with-videos framework that jointly models video grounding and question answering. VideoTemp-o3 exhibits strong localization capability, supports on-demand clipping, and can refine inaccurate localizations. Specifically, in the supervised fine-tuning stage, we design a unified masking mechanism that encourages exploration while preventing noise. For reinforcement learning, we introduce dedicated rewards to mitigate reward hacking. Besides, from the data perspective, we develop an effective pipeline to construct high-quality long video grounded QA data, along with a corresponding benchmark for systematic evaluation across various video durations. Experimental results demonstrate that our method achieves remarkable performance on both long video understanding and grounding.
Abstract (translated)
在长视频理解领域，传统的均匀帧采样方法常常无法捕捉到关键的视觉证据，从而导致性能下降和幻觉增多（即系统生成不符合事实的答案）。为了解决这一问题，最近出现了一些基于代理思考与视频处理的新范式，这些方法采用了一种“定位-剪辑-回答”的工作流程，在该流程中，模型主动识别相关的视频片段，并在这些片段内进行密集采样，最后给出答案。然而，现有的方法仍然存在效率低下、定位能力弱以及依赖固定的工作流程等问题。 为了应对这些问题，我们提出了VideoTemp-o3，这是一种统一的代理视频思考框架，能够同时建模视频基础设定（即确定视频中哪些部分与问题相关）和问答任务。VideoTemp-o3展现了强大的定位能力，并支持按需剪辑，还可以修正不准确的定位结果。具体而言，在监督微调阶段，我们设计了一种统一的掩码机制以鼓励探索并防止噪声干扰。在强化学习方面，我们引入了专门的奖励机制来减轻奖励篡改的问题。 此外，从数据角度来看，我们开发了一个有效的流程来构建高质量的长视频基础设定问答数据集，并且建立了一个对应的基准测试系统，用于评估不同长度视频的综合性能。 实验结果表明，我们的方法在长视频理解和基础设定任务上都取得了显著的效果。
URL
https://arxiv.org/abs/2602.07801
