Abstract
Long video understanding is challenging due to rich and complicated multimodal clues in long temporal this http URL methods adopt reasoning to improve the model's ability to analyze complex video clues in long videos via text-form this http URL,the existing literature suffers from the fact that the text-only reasoning under fixed video context may exacerbate hallucinations since detailed crucial clues are often ignored under limited video context length due to the temporal redundancy of long this http URL address this gap,we propose Video-TwG,a curriculum reinforced framework that employs a novel Think-with-Grounding paradigm,enabling video LLMs to actively decide when to perform on-demand grounding during interleaved text-video reasoning, selectively zooming into question-relevant clips only when this http URL-TwG can be trained end-to-end in a straightforward manner, without relying on complex auxiliary modules or heavily annotated reasoning tracesIn detail,we design a Two-stage Reinforced Curriculum Strategy, where the model first learns think-with-grounding behavior on a small short-video GQA dataset with grounding labels,and then scales to diverse general QA data with videos of diverse domains to encourage generalization. Further, to handle complex think-with-grounding reasoning for various kinds of data,we propose TwG-GRPO algorithm which features the fine-grained grounding reward, self-confirmed pseudo reward and accuracy-gated this http URL,we propose to construct a new TwG-51K dataset that facilitates training. Experiments on Video-MME, LongVideoBench, and MLVU show that Video-TwG consistently outperforms strong LVU this http URL ablation validates the necessity of our Two-stage Reinforced Curriculum Strategy and shows our TwG-GRPO better leverages diverse unlabeled data to improve grounding quality and reduce redundant groundings without sacrificing QA performance.
Abstract (translated)
长视频理解是一项挑战，因为其中包含丰富且复杂的多模态线索。现有方法通过采用推理来增强模型分析长时间视频复杂线索的能力，并将这种能力表示为文本形式的逻辑表达。然而，现有的文献存在一个问题：仅依赖固定视频上下文进行文本推理可能会加剧幻觉现象，因为在有限的视频上下文中，往往忽略了一些详细的、关键性的线索，而这些线索在冗长的视频中是至关重要的。 为了填补这一空白，我们提出了Video-TwG（Think-with-Grounding 视频框架），这是一个采用新型思考与接地范式的课程强化框架。该框架使视频LLM能够在交错的文字-视频推理过程中主动决定何时进行按需接地操作，并在仅针对问题相关片段时执行此操作。 Video-TwG可以以端到端的简便方式训练，无需依赖复杂的辅助模块或密集标注的推理轨迹。具体而言，我们设计了一种两阶段强化课程策略：模型首先在一个小规模的短视频问答数据集上学习思考与接地行为，并在此过程中使用了接地标签；随后扩展至涵盖不同领域多种类型的通用问答数据，以鼓励泛化能力。 为应对各种类型数据复杂的思考-接地推理问题，我们提出了TwG-GRPO算法，该算法具有细粒度的接地奖励、自我确认伪奖励和准确率门控机制的特点。此外，为了便于训练，我们提议构建一个新的TwG-51K数据集。 在Video-MME, LongVideoBench 和 MLVU上的实验表明，Video-TwG在长视频理解方面持续超越强大的LVU方法。消融分析验证了我们的两阶段强化课程策略的必要性，并显示了我们的TwG-GRPO算法能够更好地利用多样化未标记数据来提高接地质量并减少冗余的接地操作，同时不牺牲问答性能。 这种方法通过一种创新的方式解决了长视频理解中的核心问题，即如何在复杂的多模态环境中有效地提取和处理关键信息。
URL
https://arxiv.org/abs/2602.18702
