Abstract
Multi-modal Large Language Models (MLLMs) show promise in video understanding. However, their reasoning often suffers from thinking drift and weak temporal comprehension, even when enhanced by Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO). Moreover, existing RL methods usually depend on Supervised Fine-Tuning (SFT), which requires costly Chain-of-Thought (CoT) annotation and multi-stage training, and enforces fixed reasoning paths, limiting MLLMs' ability to generalize and potentially inducing bias. To overcome these limitations, we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT by utilizing a Structured CoT format: Summarize -> Think -> Answer. SDRL introduces two self-supervised mechanisms integrated into the GRPO objective: 1) Consistency of Vision Knowledge (CVK) enforces factual grounding by reducing KL divergence among generated summaries; and 2) Dynamic Variety of Reasoning (DVR) promotes exploration by dynamically modulating thinking diversity based on group accuracy. This novel integration effectively balances alignment and exploration, supervising both the final answer and the reasoning process. Our method achieves state-of-the-art performance on seven public VideoQA datasets.
Abstract (translated)
多模态大语言模型（MLLMs）在视频理解领域展现出潜力。然而，即便采用强化学习（RL）技术（如组相对策略优化，GRPO）进行增强，其推理过程仍常受思维漂移和时序理解能力薄弱的影响。此外，现有RL方法通常依赖监督微调（SFT），这需要成本高昂的思维链（CoT）标注和多阶段训练，并强制固定推理路径，从而限制了MLLMs的泛化能力，可能引入偏差。为克服这些局限，我们提出了摘要驱动强化学习（SDRL），一种新颖的单阶段RL框架，通过采用“摘要→思考→回答”的结构化CoT格式，消除了对SFT的需求。SDRL在GRPO目标中引入了两种自监督机制：1）视觉知识一致性（CVK）通过降低生成摘要间的KL散度来强化事实基础；2）推理动态多样性（DVR）基于组准确率动态调节思维多样性以促进探索。这种创新整合有效平衡了对齐与探索，同时监督最终答案与推理过程。我们的方法在七个公开视频问答数据集上达到了最先进性能。
URL
https://arxiv.org/abs/2603.25942
