Video_Caption
-
A Benchmarking Methodology to Assess Open-Source Video Large Language Models in Automatic Captioning of News Videos
2026-03-29 12:28:35David Miranda Paredes, Jose M. Saavedra, Marcelo PizarroAbstract
News videos are among the most prevalent content types produced by television stations and online streaming platforms, yet generating textual descriptions to facilitate indexing and retrieval largely remains a manual process. Video Large Language Models (VidLLMs) offer significant potential to automate this task, but a comprehensive evaluation in the news domain is still lacking. This work presents a comparative study of eight state-of-the-art open-source VidLLMs for automatic news video captioning, evaluated on two complementary benchmark datasets: a Chilean TV news corpus (approximately 1,345 clips) and a BBC News corpus (9,838 clips). We employ lexical metrics (METEOR, ROUGE-L), semantic metrics (BERTScore, CLIPScore, Text Similarity, Mean Reciprocal Rank), and two novel fidelity metrics proposed in this work: the Thematic Fidelity Score (TFS) and Entity Fidelity Score (EFS). Our analysis reveals that standard metrics exhibit limited discriminative power for news video captioning due to surface-form dependence, static-frame insensitivity, and function-word inflation. TFS and EFS address these gaps by directly assessing thematic structure preservation and named-entity coverage in the generated captions. Results show that Gemma~3 achieves the highest overall performance across both datasets and most evaluation dimensions, with Qwen-VL as a consistent runner-up.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27662
PDF
-
V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models
2026-03-29 11:53:32Xinying Lin, Xuyang Liu, Yiyu Wang, Teng Ma, Wenqi RenAbstract
Video large language models (VideoLLMs) show strong capability in video understanding, yet long-context inference is still dominated by massive redundant visual tokens in the prefill stage. We revisit token compression for VideoLLMs under a tight budget and identify a key bottleneck, namely insufficient spatio-temporal information coverage. Existing methods often introduce discontinuous coverage through coarse per-frame allocation or scene segmentation, and token merging can further misalign spatio-temporal coordinates under MRoPE-style discrete (t,h,w) bindings. To address these issues, we propose V-CAST (Video Curvature-Aware Spatio-Temporal Pruning), a training-free, plug-and-play pruning policy for long-context video inference. V-CAST casts token compression as a trajectory approximation problem and introduces a curvature-guided temporal allocation module that routes per-frame token budgets to semantic turns and event boundaries. It further adopts a dual-anchor spatial selection mechanism that preserves high-entropy visual evidence without attention intervention, while keeping retained tokens at their original coordinates to maintain positional alignment. Extensive experiments across multiple VideoLLMs of different architectures and scales demonstrate that V-CAST achieves 98.6% of the original performance, outperforms the second-best method by +1.1% on average, and reduces peak memory and total latency to 86.7% and 86.4% of vanilla Qwen3-VL-8B-Instruct.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27650
PDF
-
STRIDE: When to Speak Meets Sequence Denoising for Streaming Video Understanding
2026-03-29 09:23:45Junho Kim, Hosu Lee, James M. Rehg, Minsu Kim, Yong Man RoAbstract
Recent progress in video large language models (Video-LLMs) has enabled strong offline reasoning over long and complex videos. However, real-world deployments increasingly require streaming perception and proactive interaction, where video frames arrive online and the system must decide not only what to respond, but also when to respond. In this work, we revisit proactive activation in streaming video as a structured sequence modeling problem, motivated by the observation that temporal transitions in streaming video naturally form span-structured activation patterns. To capture this span-level structure, we model activation signals jointly over a sliding temporal window and update them iteratively as new frames arrive. We propose STRIDE (Structured Temporal Refinement with Iterative DEnoising), which employs a lightweight masked diffusion module at the activation interface to jointly predict and progressively refine activation signals across the window. Extensive experiments on diverse streaming benchmarks and downstream models demonstrate that STRIDE shows more reliable and temporally coherent proactive responses, significantly improving when-to-speak decision quality in online streaming scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27593
PDF
-
Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark
2026-03-28 12:44:19Seng Nam Chen, Hao Chen, Chenglam Ho, Xinyu Mao, Jinping Wang, Yu Zhang, Chao LiAbstract
Long video understanding (LVU) remains a core challenge in multimodal learning. Although recent vision-language models (VLMs) have made notable progress, existing benchmarks mainly focus on either fine-grained perception or coarse summarization, offering limited insight into temporal understanding over long contexts. In this work, we define a scene as a coherent segment of a video in which both visual and semantic contexts remain consistent, aligning with human perception. This leads us to a key question: can current VLMs reason effectively over long, scene-level contexts? To answer this, we introduce a new benchmark, SceneBench, designed to provide scene-level challenges. Our evaluation reveals a sharp drop in accuracy when VLMs attempt to answer scene-level questions, indicating significant forgetting of long-range context. To further validate these findings, we propose Scene Retrieval-Augmented Generation (Scene-RAG), which constructs a dynamic scene memory by retrieving and integrating relevant context across scenes. This Scene-RAG improves VLM performance by +2.50%, confirming that current models still struggle with long-context retention. We hope SceneBench will encourage future research toward VLMs with more robust, human-like video comprehension.
Abstract (translated)
URL
https://arxiv.org/abs/2603.27259
PDF
-
Incentivizing Temporal-Awareness in Egocentric Video Understanding Models
2026-03-28 08:02:59Zhiyang Xu, Tian Qin, Bowen Jin, Zhengfeng Lai, Meng Cao, Lifu Huang, Peng ZhangAbstract
Multimodal large language models (MLLMs) have recently shown strong performance in visual understanding, yet they often lack temporal awareness, particularly in egocentric settings where reasoning depends on the correct ordering and evolution of events. This deficiency stems in part from training objectives that fail to explicitly reward temporal reasoning and instead rely on frame-level spatial shortcuts. To address this limitation, we propose Temporal Global Policy Optimization (TGPO), a reinforcement learning with verifiable rewards (RLVR) algorithm designed to incentivize temporal awareness in MLLMs. TGPO contrasts model outputs generated from temporally ordered versus shuffled video frames to derive calibrated, globally normalized reward signals that explicitly favor temporally coherent reasoning. Integrated with GRPO and GSPO, TGPO supports cold-start RL training and effectively suppresses spatial shortcut behaviors learned by existing MLLMs. Experiments across five egocentric video benchmarks demonstrate that TGPO consistently improves temporal grounding and causal coherence, outperforming prior RL-based video reasoning approaches. Our results suggest that TGPO offers a simple and scalable pathway toward temporally robust MLLMs for egocentric video understanding.
Abstract (translated)
多模态大语言模型（MLLMs）近期在视觉理解方面展现出强劲性能，但往往缺乏时序感知能力，尤其在依赖事件正确排序与演变的第一人称视角场景中。这一缺陷部分源于训练目标未能明确奖励时序推理，而仅依赖帧级空间捷径。为应对此局限，我们提出时序全局策略优化（TGPO）——一种基于可验证奖励的强化学习（RLVR）算法，旨在激发MLLMs的时序感知能力。TGPO通过对比模型对时序有序与打乱视频帧的输出，生成经校准的全局归一化奖励信号，明确优先选择时序连贯的推理。TGPO与GRPO及GSPO相结合，支持冷启动强化学习训练，并有效抑制现有MLLMs习得的空间捷径行为。在五个第一人称视频基准测试上的实验表明，TGPO持续提升时序定位与因果连贯性，优于此前基于强化学习的视频推理方法。研究结果表明，TGPO为构建面向第一人称视频理解的时序鲁棒性MLLMs提供了简单且可扩展的路径。
URL
https://arxiv.org/abs/2603.27184
PDF
-
Make Geometry Matter for Spatial Reasoning
2026-03-27 17:45:12Shihua Zhang, Qiuhong Shen, Shizun Wang, Tianbo Pan, Xinchao WangAbstract
Empowered by large-scale training, vision-language models (VLMs) achieve strong image and video understanding, yet their ability to perform spatial reasoning in both static scenes and dynamic videos remains limited. Recent advances try to handle this limitation by injecting geometry tokens from pretrained 3D foundation models into VLMs. Nevertheless, we observe that naive token fusion followed by standard fine-tuning in this line of work often leaves such geometric cues underutilized for spatial reasoning, as VLMs tend to rely heavily on 2D visual cues. In this paper, we propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry tokens. GeoSR introduces two key components: (1) Geometry-Unleashing Masking, which strategically masks portions of 2D vision tokens during training to weaken non-geometric shortcuts and force the model to consult geometry tokens for spatial reasoning; and (2) Geometry-Guided Fusion, a gated routing mechanism that adaptively amplifies geometry token contributions in regions where geometric evidence is critical. Together, these designs unleash the potential of geometry tokens for spatial reasoning tasks. Extensive experiments on both static and dynamic spatial reasoning benchmarks demonstrate that GeoSR consistently outperforms prior methods and establishes new state-of-the-art performance by effectively leveraging geometric information. The project page is available at this https URL.
Abstract (translated)
在大规模训练的赋能下，视觉语言模型（VLMs）已具备强大的图像与视频理解能力，但其在静态场景与动态视频中的空间推理能力仍存在局限。近期研究尝试通过将预训练三维基础模型中的几何令牌注入VLMs来应对这一挑战。然而我们观察到，此类工作中简单的令牌融合配合标准微调往往导致几何线索在空间推理中利用不足，因为VLMs倾向于过度依赖二维视觉线索。本文提出GeoSR框架，旨在通过促使VLMs主动运用几何令牌来提升几何信息的重要性。GeoSR引入两个关键组件：（1）几何释放掩码（Geometry-Unleashing Masking），在训练过程中策略性掩蔽部分二维视觉令牌，以削弱非几何捷径并迫使模型在空间推理中 consult 几何令牌；（2）几何引导融合（Geometry-Guided Fusion），一种自适应增强几何令牌在关键几何证据区域贡献的门控路由机制。这些设计共同释放了几何令牌在空间推理任务中的潜力。在静态与动态空间推理基准测试上的大量实验表明，GeoSR通过有效利用几何信息持续优于先前方法，并建立了新的最先进性能。项目页面可通过此链接访问。
URL
https://arxiv.org/abs/2603.26639
PDF
-
Dynamic Token Compression for Efficient Video Understanding through Reinforcement Learning
2026-03-27 12:46:25Shida Wang, YongXiang Hua, Zhou Tao, Haoyu Cao, Linli XuAbstract
Multimodal Large Language Models have demonstrated remarkable capabilities in video understanding, yet face prohibitive computational costs and performance degradation from ''context rot'' due to massive visual token redundancy. Existing compression strategies typically rely on heuristics or fixed transformations that are often decoupled from the downstream task objectives, limiting their adaptability and effectiveness. To address this, we propose SCORE (Surprise-augmented token COmpression via REinforcement learning), a unified framework that learns an adaptive token compression policy. SCORE introduces a lightweight policy network conditioned on a surprise-augmented state representation that incorporates inter-frame residuals to explicitly capture temporal dynamics and motion saliency. We optimize this policy using a group-wise reinforcement learning scheme with a split-advantage estimator, stabilized by a two-stage curriculum transferring from static pseudo-videos to real dynamic videos. Extensive experiments on diverse video understanding benchmarks demonstrate that SCORE significantly outperforms state-of-the-art baselines. Notably, SCORE achieves a 16x prefill speedup while preserving 99.5% of original performance at a 10% retention ratio, offering a scalable solution for efficient long-form video understanding.
Abstract (translated)
多模态大语言模型在视频理解方面展现出卓越能力，但面临因海量视觉标记冗余导致的巨大计算成本及“上下文衰减”引起的性能下降。现有压缩策略通常依赖启发式方法或固定变换，且常与下游任务目标脱节，限制了其适应性和有效性。针对此问题，我们提出SCORE（基于增强惊讶度的强化学习标记压缩）统一框架，该框架可学习自适应标记压缩策略。SCORE引入轻量级策略网络，其条件基于融合帧间残差的增强惊讶度状态表征，以显式捕捉时间动态与运动显著性。我们采用分组强化学习方案结合分裂优势估计器优化该策略，并通过从静态伪视频到真实动态视频的两阶段课程迁移实现稳定。在多种视频理解基准测试上的广泛实验表明，SCORE显著优于最先进基线方法。值得注意的是，在保留10%标记比例时，SCORE实现16倍预填充加速同时保持99.5%原始性能，为高效长视频理解提供了可扩展解决方案。
URL
https://arxiv.org/abs/2603.26365
PDF
-
From Pixels to Privacy: Temporally Consistent Video Anonymization via Token Pruning for Privacy Preserving Action Recognition
2026-03-27 11:57:51Nazia Aslam, Abhisek Ray, Joakim Bruslund Haurum, Lukas Esterle, Kamal NasrollahiAbstract
Recent advances in large-scale video models have significantly improved video understanding across domains such as surveillance, healthcare, and entertainment. However, these models also amplify privacy risks by encoding sensitive attributes, including facial identity, race, and gender. While image anonymization has been extensively studied, video anonymization remains relatively underexplored, even though modern video models can leverage spatiotemporal motion patterns as biometric identifiers. To address this challenge, we propose a novel attention-driven spatiotemporal video anonymization framework based on systematic disentanglement of utility and privacy features. Our key insight is that attention mechanisms in Vision Transformers (ViTs) can be explicitly structured to separate action-relevant information from privacy-sensitive content. Building on this insight, we introduce two task-specific classification tokens, an action CLS token and a privacy CLS token, that learn complementary representations within a shared Transformer backbone. We contrast their attention distributions to compute a utility-privacy score for each spatiotemporal tubelet, and keep the top-k tubelets with the highest scores. This selectively prunes tubelets dominated by privacy cues while preserving those most critical for action recognition. Extensive experiments demonstrate that our approach maintains action recognition performance comparable to models trained on raw videos, while substantially reducing privacy leakage. These results indicate that attention-driven spatiotemporal pruning offers an effective and principled solution for privacy-preserving video analytics.
Abstract (translated)
大规模视频模型的最新进展显著提升了监控、医疗和娱乐等领域的视频理解能力。然而，这些模型也通过编码面部身份、种族和性别等敏感属性而放大了隐私风险。尽管图像匿名化技术已得到广泛研究，视频匿名化仍相对缺乏探索——而现代视频模型能够利用时空运动模式作为生物特征标识符。为应对这一挑战，我们提出了一种基于效用与隐私特征系统性解耦的、新型注意力驱动时空视频匿名化框架。我们的核心见解是：视觉Transformer（ViTs）中的注意力机制可被显式结构化，以分离与动作相关的信息与隐私敏感内容。基于此见解，我们引入了两个任务特定的分类标记——动作CLS标记与隐私CLS标记，使它们能在共享Transformer主干网络中学习互补表征。我们通过对比它们的注意力分布，为每个时空小片段计算效用-隐私分数，并保留分数最高的前k个小片段。这种方法能选择性地剪除由隐私线索主导的小片段，同时保留对动作识别最关键的部分。大量实验表明，我们的方法在保持与原始视频训练模型相当的动作识别性能的同时，显著降低了隐私泄露风险。这些结果表明，注意力驱动的时空剪枝为隐私保护视频分析提供了有效且 principled 的解决方案。
URL
https://arxiv.org/abs/2603.26336
PDF
-
Reinforcing Structured Chain-of-Thought for Video Understanding
2026-03-26 22:11:14Peiyao Wang, Haotian Xu, Noranart Vesdapunt, Rui Hou, Jingyi Zhang, Haibin Ling, Oleksandr Obiednikov, Ning Zhou, Kah Kuen FuAbstract
Multi-modal Large Language Models (MLLMs) show promise in video understanding. However, their reasoning often suffers from thinking drift and weak temporal comprehension, even when enhanced by Reinforcement Learning (RL) techniques like Group Relative Policy Optimization (GRPO). Moreover, existing RL methods usually depend on Supervised Fine-Tuning (SFT), which requires costly Chain-of-Thought (CoT) annotation and multi-stage training, and enforces fixed reasoning paths, limiting MLLMs' ability to generalize and potentially inducing bias. To overcome these limitations, we introduce Summary-Driven Reinforcement Learning (SDRL), a novel single-stage RL framework that obviates the need for SFT by utilizing a Structured CoT format: Summarize -> Think -> Answer. SDRL introduces two self-supervised mechanisms integrated into the GRPO objective: 1) Consistency of Vision Knowledge (CVK) enforces factual grounding by reducing KL divergence among generated summaries; and 2) Dynamic Variety of Reasoning (DVR) promotes exploration by dynamically modulating thinking diversity based on group accuracy. This novel integration effectively balances alignment and exploration, supervising both the final answer and the reasoning process. Our method achieves state-of-the-art performance on seven public VideoQA datasets.
Abstract (translated)
多模态大语言模型（MLLMs）在视频理解领域展现出潜力。然而，即便采用强化学习（RL）技术（如组相对策略优化，GRPO）进行增强，其推理过程仍常受思维漂移和时序理解能力薄弱的影响。此外，现有RL方法通常依赖监督微调（SFT），这需要成本高昂的思维链（CoT）标注和多阶段训练，并强制固定推理路径，从而限制了MLLMs的泛化能力，可能引入偏差。为克服这些局限，我们提出了摘要驱动强化学习（SDRL），一种新颖的单阶段RL框架，通过采用“摘要→思考→回答”的结构化CoT格式，消除了对SFT的需求。SDRL在GRPO目标中引入了两种自监督机制：1）视觉知识一致性（CVK）通过降低生成摘要间的KL散度来强化事实基础；2）推理动态多样性（DVR）基于组准确率动态调节思维多样性以促进探索。这种创新整合有效平衡了对齐与探索，同时监督最终答案与推理过程。我们的方法在七个公开视频问答数据集上达到了最先进性能。
URL
https://arxiv.org/abs/2603.25942
PDF
-
GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding
2026-03-26 19:03:49Trong Thang Pham, Hien Nguyen, Ngan LeAbstract
Current multimodal large language models (MLLMs) cannot effectively utilize eye-gaze information for video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler (~1-5 M trainable parameters) that encodes V-JEPA 2.1 video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze benchmark, GazeQwen reaches 63.9% accuracy, a +16.1 point gain over the same Qwen2.5-VL-7B backbone with gaze as visual prompts and +10.5 points over GPT-4o, the highest score among all open-source and proprietary models tested. These results suggest that learning where to inject gaze within an LLM is more effective than scaling model size or engineering better prompts. All code and checkpoints are available at this https URL .
Abstract (translated)
当前多模态大语言模型（MLLMs）无法有效利用眼动信息进行视频理解，即使通过视觉叠加或文本描述提供眼动线索。我们提出GazeQwen，一种参数高效的方法，通过隐状态调制为开源MLLM赋予眼动感知能力。其核心是一个紧凑的眼动重采样器（约1-5M可训练参数），该模块编码V-JEPA 2.1视频特征，结合注视点衍生的位置编码，并通过前向钩子将生成的可加残差注入选定的LLM解码器层。可选的第二阶段训练会为LLM添加低秩适配器（LoRA）以实现更紧密的集成。在StreamGaze基准的全部10项任务评估中，GazeQwen达到63.9%的准确率，相比使用眼动作为视觉提示的相同Qwen2.5-VL-7B主干网络提升16.1个百分点，并超越GPT-4o（所有测试的开源及闭源模型中最高分）10.5个百分点。这些结果表明，学习在LLM中注入眼动信息的位置比扩大模型规模或设计更优提示更有效。所有代码与检查点可通过此https URL获取。
URL
https://arxiv.org/abs/2603.25841
PDF
-
Learning to Rank Caption Chains for Video-Text Alignment
2026-03-26 08:04:57Ansel Blume, Burak Uzkent, Shalini Chaudhuri, Garin KesslerAbstract
Direct preference optimization (DPO) is an effective technique to train language models to generate preferred over dispreferred responses. However, this binary "winner-takes-all" approach is suboptimal for vision-language models whose response quality is highly dependent on visual content. In particular, a response may still be faithful to the visual inputs even if it is less preferable than an alternative. The standard Bradley-Terry DPO formulation lacks this nuance, upweighting winning responses without sufficient regard for whether the "losing" response still maintains high visual fidelity. In this work, we investigate ranking optimization as an alternative that more precisely situates responses' faithfulness to visual inputs. We focus on video-text alignment using detailed video captions, proposing a method to generate challenging, totally ordered caption chains at scale through repeated caption degradation. Our results show ranking optimization outperforms binary DPO for long-form content generation and assessment, and importantly, we find that these approaches require finetuning of the vision encoder to be effective, challenging the view of DPO as purely a language-reweighting process.
Abstract (translated)
直接偏好优化（DPO）是一种有效的技术，用于训练语言模型以生成优于次优响应的内容。然而，这种二元“赢家通吃”的方法对于视觉-语言模型而言并非最优，因为后者的响应质量高度依赖于视觉内容。具体而言，即使某个响应不如替代方案理想，它仍可能对视觉输入保持忠实。标准的Bradley-Terry DPO公式缺乏这种细微差别，它在过度加权获胜响应的同时，未充分关注“失败”响应是否仍维持较高的视觉保真度。在本研究中，我们探索将排序优化作为替代方案，以更精确地定位响应与视觉输入之间的忠实度关系。我们聚焦于使用详细视频字幕的视频-文本对齐任务，提出了一种通过重复字幕降解来大规模生成具有挑战性的完全有序字幕链的方法。结果表明，排序优化在长内容生成与评估中优于二元DPO。更重要的是，我们发现这些方法需要微调视觉编码器才能发挥效果，这挑战了将DPO视为纯粹语言重加权过程的观点。
URL
https://arxiv.org/abs/2603.25145
PDF
-
GIFT: Global Irreplaceability Frame Targeting for Efficient Video Understanding
2026-03-26 06:21:41Junpeng Ma, Sashuai Zhou, Guanghao Li, Xin Gao, Yue Cao, Hengyu Zeng, Yuxiang Yan, Zhibin Wang, Jun Song, Bo Zheng, Shanghang Zhang, Jian PuAbstract
Video Large Language Models (VLMs) have achieved remarkable success in video understanding, but the significant computational cost from processing dense frames severely limits their practical application. Existing methods alleviate this by selecting keyframes, but their greedy decision-making, combined with a decoupled evaluation of relevance and diversity, often falls into local optima and results in erroneously selecting irrelevant noise frames. To address these challenges, we propose GIFT: Global Irreplaceability Frame Targeting, a novel training-free framework that selects frames by assessing their intrinsic irreplaceability. Specifically, we first introduce Directed Diversity to quantify a frame's uniqueness conditioned on relevance, which allows us to formulate a unified irreplaceability score. Subsequently, our Budget-Aware Refinement strategy employs a adaptive iterative process that first secures a core set of frames with the highest irreplaceability, and then shifts its priority to building crucial temporal context around these selections as the budget expands. Extensive experiments demonstrate that GIFT achieves a maximum average improvement of 12.5% across long-form video benchmarks on LLaVA-Video-7B compared to uniform sampling.
Abstract (translated)
视频大语言模型（VLMs）在视频理解领域已取得显著成功，但处理密集帧所带来的巨大计算成本严重制约了其实际应用。现有方法通过选择关键帧来缓解此问题，然而其贪婪决策结合对相关性和多样性的解耦评估，常陷入局部最优并导致错误选择无关噪声帧。为应对这些挑战，我们提出GIFT：全局不可替代帧定位，一种新颖的无训练框架，通过评估帧的内在不可替代性进行选择。具体而言，我们首先引入导向多样性（Directed Diversity）来量化在相关性条件下的帧独特性，从而构建统一的不可替代性评分。随后，我们的预算感知优化策略（Budget-Aware Refinement）采用自适应迭代过程：首先锁定不可替代性最高的核心帧集，随后随预算扩展将优先级转向围绕这些选择构建关键时序上下文。大量实验表明，在长视频基准测试中，GIFT在LLaVA-Video-7B上相比均匀采样实现了最高12.5%的平均提升。
URL
https://arxiv.org/abs/2603.25072
PDF
-
VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning
2026-03-26 04:37:54Zhe Gao, Shiyu Shen, Taifeng Chai, Weinong Wang, Haotian Xu, Xing W, Wenbin Li, Qi Fan, Yang Gao, Dacheng TaoAbstract
Existing Multimodal Large Language Models (MLLMs) often suffer from hallucinations in long video understanding (LVU), primarily due to the imbalance between textual and visual tokens. Observing that MLLMs handle short visual inputs well, recent LVU works alleviate hallucinations by automatically parsing the vast visual data into manageable segments that can be effectively processed by MLLMs. SFT-based tool-calling methods can serve this purpose, but they typically require vast amounts of fine-grained, high-quality data and suffer from constrained tool-calling trajectories. We propose a novel VideoTIR that leverages Reinforcement Learning (RL) to encourage proper usage of comprehensive multi-level toolkits for efficient long video understanding. VideoTIR explores both Zero-RL and SFT cold-starting to enable MLLMs to retrieve and focus on meaningful video segments/images/regions, enhancing long video understanding both accurately and efficiently. To reduce redundant tool-calling, we propose Toolkit Action Grouped Policy Optimization (TAGPO), which enhances the efficiency of the calling process through stepwise reward assignment and reuse of failed rollouts. Additionally, we develop a sandbox-based trajectory synthesis framework to generate high-quality trajectories data. Extensive experiments on three long-video QA benchmarks demonstrate the effectiveness and efficiency of our method.
Abstract (translated)
现有多模态大语言模型（MLLMs）在长视频理解（LVU）中常因文本与视觉标记的失衡而产生幻觉。鉴于MLLMs能有效处理短视觉输入，近期LVU研究通过自动解析海量视觉数据为可管理片段来缓解幻觉，从而让MLLMs有效处理。基于监督微调（SFT）的工具调用方法虽可达成此目的，但通常需要大量细粒度高质量数据，且受限于工具调用轨迹。我们提出一种新颖的VideoTIR方法，利用强化学习（RL）促进对多层次综合工具集的合理使用，以实现高效长视频理解。VideoTIR探索零样本RL与SFT冷启动两种方式，使MLLMs能检索并聚焦于有意义的视频片段/图像/区域，从而更准确高效地增强长视频理解。为减少冗余工具调用，我们提出工具操作组策略优化（TAGPO），通过逐步奖励分配与失败轨迹复用提升调用效率。此外，我们开发了基于沙箱的轨迹合成框架以生成高质量轨迹数据。在三个长视频问答基准测试上的大量实验验证了我们方法的有效性与高效性。
URL
https://arxiv.org/abs/2603.25021
PDF
-
LensWalk: Agentic Video Understanding by Planning How You See in Videos
2026-03-25 17:38:54Keliang Li, Yansong Li, Hongze Shen, Mengdi Liu, Hong Chang, Shiguang ShanAbstract
The dense, temporal nature of video presents a profound challenge for automated analysis. Despite the use of powerful Vision-Language Models, prevailing methods for video understanding are limited by the inherent disconnect between reasoning and perception: they rely on static, pre-processed information and cannot actively seek raw evidence from video as their understanding evolves. To address this, we introduce LensWalk, a flexible agentic framework that empowers a Large Language Model reasoner to control its own visual observation actively. LensWalk establishes a tight reason-plan-observe loop where the agent dynamically specifies, at each step, the temporal scope and sampling density of the video it observes. Using a suite of versatile, Vision-Language Model based tools parameterized by these specifications, the agent can perform broad scans for cues, focus on specific segments for fact extraction, and stitch evidence from multiple moments for holistic verification. This design allows for progressive, on-demand evidence gathering that directly serves the agent's evolving chain of thought. Without requiring any model fine-tuning, LensWalk delivers substantial, plug-and-play performance gains on multiple model recipes, boosting their accuracy by over 5\% on challenging long-video benchmarks like LVBench and Video-MME. Our analysis reveals that enabling an agent to control how it sees is key to unlocking more accurate, robust, and interpretable video reasoning.
Abstract (translated)
视频密集的时序特性给自动化分析带来了深刻挑战。尽管使用了强大的视觉-语言模型，当前视频理解方法仍受限于推理与感知之间的固有脱节：它们依赖静态的预处理信息，无法在理解演进过程中主动从视频中寻求原始证据。为解决这一问题，我们提出了LensWalk——一个灵活的智能体框架，使大语言模型推理器能够主动控制自身的视觉观察。LensWalk建立了紧密的推理-规划-观察循环，智能体在每一步动态指定其观察视频的时间范围和采样密度。通过一套由这些参数控制的多功能视觉-语言模型工具，智能体能够进行广泛扫描以寻找线索，聚焦特定片段提取事实，并整合多时刻证据进行整体验证。这一设计支持了渐进式、按需的证据收集，直接服务于智能体不断演进的思维链。LensWalk无需任何模型微调，即可在多种模型配置上实现显著的即插即用性能提升，在LVBench和Video-MME等具有挑战性的长视频基准上将准确率提高了超过5%。我们的分析表明，使智能体能够自主控制观察方式是实现更准确、更稳健、更具可解释性的视频推理的关键。
URL
https://arxiv.org/abs/2603.24558
PDF
-
CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition
2026-03-25 17:14:36Florian Stilz, Vinkle Srivastav, Nassir Navab, Nicolas PadoyAbstract
Video-language foundation models have proven to be highly effective in zero-shot applications across a wide range of tasks. A particularly challenging area is the intraoperative surgical procedure domain, where labeled data is scarce, and precise temporal understanding is often required for complex downstream tasks. To address this challenge, we introduce CliPPER (Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition), a novel video-language pretraining framework trained on surgical lecture videos. Our method is designed for fine-grained temporal video-text recognition and introduces several novel pretraining strategies to improve multimodal alignment in long-form surgical videos. Specifically, we propose Contextual Video-Text Contrastive Learning (VTC_CTX) and Clip Order Prediction (COP) pretraining objectives, both of which leverage temporal and contextual dependencies to enhance local video understanding. In addition, we incorporate a Cycle-Consistency Alignment over video-text matches within the same surgical video to enforce bidirectional consistency and improve overall representation coherence. Moreover, we introduce a more refined alignment loss, Frame-Text Matching (FTM), to improve the alignment between video frames and text. As a result, our model establishes a new state-of-the-art across multiple public surgical benchmarks, including zero-shot recognition of phases, steps, instruments, and triplets. The source code and pretraining captions can be found at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.24539
PDF
-
GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents
2026-03-25 14:10:45Yunzhe Wang, Runhui Xu, Kexin Zheng, Tianyi Zhang, Jayavibhav Niranjan Kogundi, Soham Hans, Volkan UstunAbstract
Multimodal LLMs are increasingly deployed as perceptual backbones for autonomous agents in 3D environments, from robotics to virtual worlds. These applications require agents to perceive rapid state changes, attribute actions to the correct entities, and reason about concurrent multi-agent behaviors from a first-person perspective, capabilities that existing benchmarks do not adequately evaluate. We introduce GameplayQA, a framework for evaluating agentic-centric perception and reasoning through video understanding. Specifically, we densely annotate multiplayer 3D gameplay videos at 1.22 labels/second, with time-synced, concurrent captions of states, actions, and events structured around a triadic system of Self, Other Agents, and the World, a natural decomposition for multi-agent environments. From these annotations, we refined 2.4K diagnostic QA pairs organized into three levels of cognitive complexity, accompanied by a structured distractor taxonomy that enables fine-grained analysis of where models hallucinate. Evaluation of frontier MLLMs reveals a substantial gap from human performance, with common failures in temporal and cross-video grounding, agent-role attribution, and handling the decision density of the game. We hope GameplayQA stimulates future research at the intersection of embodied AI, agentic perception, and world modeling.
Abstract (translated)
URL
https://arxiv.org/abs/2603.24329
PDF
-
From Content to Audience: A Multimodal Annotation Framework for Broadcast Television Analytics
2026-03-24 13:46:51Paolo Cupini, Francesco PierriAbstract
Automated semantic annotation of broadcast television content presents distinctive challenges, combining structured audiovisual composition, domain-specific editorial patterns, and strict operational constraints. While multimodal large language models (MLLMs) have demonstrated strong general-purpose video understanding capabilities, their comparative effectiveness across pipeline architectures and input configurations in broadcast-specific settings remains empirically undercharacterized. This paper presents a systematic evaluation of multimodal annotation pipelines applied to broadcast television news in the Italian setting. We construct a domain-specific benchmark of clips labeled across four semantic dimensions: visual environment classification, topic classification, sensitive content detection, and named entity recognition. Two different pipeline architectures are evaluated across nine frontier models, including Gemini 3.0 Pro, LLaMA 4 Maverick, Qwen-VL variants, and Gemma 3, under progressively enriched input strategies combining visual signals, automatic speech recognition, speaker diarization, and metadata. Experimental results demonstrate that gains from video input are strongly model-dependent: larger models effectively leverage temporal continuity, while smaller models show performance degradation under extended multimodal context, likely due to token overload. Beyond benchmarking, the selected pipeline is deployed on 14 full broadcast episodes, with minute-level annotations integrated with normalized audience measurement data provided by an Italian media company. This integration enables correlational analysis of topic-level audience sensitivity and generational engagement divergence, demonstrating the operational viability of the proposed framework for content-based audience analytics.
Abstract (translated)
广播电视内容的自动化语义标注面临独特挑战，需综合处理结构化的视听构成、领域特定的编辑模式以及严格的操作限制。尽管多模态大语言模型（MLLMs）已展现出强大的通用视频理解能力，但针对广播电视特定场景下不同流水线架构与输入配置的相对有效性，尚缺乏实证研究。本文系统评估了应用于意大利语境下电视新闻的多模态标注流水线。我们构建了涵盖四个语义维度的领域专用基准数据集：视觉环境分类、话题分类、敏感内容检测与命名实体识别。实验在九种前沿模型（包括Gemini 3.0 Pro、LLaMA 4 Maverick、Qwen-VL系列及Gemma 3）上对比了两种流水线架构，并采用渐进式增强的输入策略，逐步融合视觉信号、自动语音识别、说话人分离与元数据。实验结果表明，视频输入带来的性能提升高度依赖模型规模：大模型能有效利用时间连续性，而小模型在扩展多模态上下文时出现性能退化，可能源于token过载。除基准测试外，选定流水线已部署于14集完整广播节目，其分钟级标注与意大利媒体公司提供的标准化收视率数据实现整合。该整合支持了话题级受众敏感度与代际参与差异的相关性分析，验证了所提框架在基于内容的受众分析中的 operational viability。
URL
https://arxiv.org/abs/2603.26772
PDF
-
Cluster-Wise Spatio-Temporal Masking for Efficient Video-Language Pretraining
2026-03-24 08:48:15Weijun Zhuang, Yuqing Huang, Weikang Meng, Xin Li, Ming Liu, Xiaopeng Hong, Yaowei Wang, Wangmeng ZuoAbstract
Large-scale video-language pretraining enables strong generalization across multimodal tasks but often incurs prohibitive computational costs. Although recent advances in masked visual modeling help mitigate this issue, they still suffer from two fundamental limitations: severe visual information loss under high masking ratios and temporal information leakage caused by inter-frame correlations. To address these challenges, we propose ClusterSTM, a Cluster-Wise Spatio-Temporal Masking strategy for efficient video-language pretraining. ClusterSTM first performs intra-frame clustering to partition visual tokens into multiple semantically independent clusters, then conducts cluster-wise masking by retaining the token with the highest temporal density within each cluster. Our masking strategy ensure that the retained tokens capture holistic video content while exhibit strong temporal correlation. Additionally, we introduce a video-text relevance reconstruction objective that aligns high-level multimodal semantics beyond conventional visual reconstruction. Extensive experiments across multiple benchmarks demonstrate that ClusterSTM achieves superior performance on video-text retrieval, video question answering, and video captioning tasks, establishing a new state-of-the-art among efficient video-language models.
Abstract (translated)
大规模视频-语言预训练虽能实现多模态任务的强泛化能力，但常伴随高昂的计算成本。尽管近期掩码视觉建模的进展有助于缓解此问题，仍存在两个根本局限：高掩码比率下严重的视觉信息损失，以及帧间相关性导致的时间信息泄露。为应对这些挑战，我们提出ClusterSTM——一种面向高效视频-语言预训练的聚类时空掩码策略。该方法首先进行帧内聚类，将视觉标记划分为多个语义独立的聚类；随后执行聚类掩码，在每个聚类中保留时间密度最高的标记。此策略确保保留的标记既能捕获视频整体内容，又具备强时间相关性。此外，我们引入视频-文本相关性重建目标，超越传统视觉重建以对齐高层多模态语义。多项基准实验表明，ClusterSTM在视频-文本检索、视频问答和视频描述任务中均取得卓越性能，在高效视频-语言模型中建立了新的最先进水平。
URL
https://arxiv.org/abs/2603.22953
PDF
-
EVA: Efficient Reinforcement Learning for End-to-End Video Agent
2026-03-24 08:06:29Yaolun Zhang, Ruohui Wang, Jiahao Wang, Yepeng Tang, Xuanyu Zheng, Haonan Duan, Hao Lu, Hanming Deng, Lewei LuAbstract
Video understanding with multimodal large language models (MLLMs) remains challenging due to the long token sequences of videos, which contain extensive temporal dependencies and redundant frames. Existing approaches typically treat MLLMs as passive recognizers, processing entire videos or uniformly sampled frames without adaptive reasoning. Recent agent-based methods introduce external tools, yet still depend on manually designed workflows and perception-first strategies, resulting in inefficiency on long videos. We present EVA, an Efficient Reinforcement Learning framework for End-to-End Video Agent, which enables planning-before-perception through iterative summary-plan-action-reflection reasoning. EVA autonomously decides what to watch, when to watch, and how to watch, achieving query-driven and efficient video understanding. To train such agents, we design a simple yet effective three-stage learning pipeline - comprising supervised fine-tuning (SFT), Kahneman-Tversky Optimization (KTO), and Generalized Reward Policy Optimization (GRPO) - that bridges supervised imitation and reinforcement learning. We further construct high-quality datasets for each stage, supporting stable and reproducible training. We evaluate EVA on six video understanding benchmarks, demonstrating its comprehensive capabilities. Compared with existing baselines, EVA achieves a substantial improvement of 6-12% over general MLLM baselines and a further 1-3% gain over prior adaptive agent methods. Our code and model are available at this https URL.
Abstract (translated)
基于多模态大语言模型（MLLMs）的视频理解仍具挑战性，因为视频包含长序列令牌，具有大量时间依赖性和冗余帧。现有方法通常将MLLMs视为被动识别器，处理整个视频或均匀采样的帧，缺乏自适应推理。近期基于智能体的方法引入外部工具，但仍依赖手动设计的工作流和感知优先策略，导致长视频处理效率低下。我们提出EVA——一个用于端到端视频智能体的高效强化学习框架，通过迭代总结-规划-行动-反思的推理机制实现“先规划后感知”。EVA能自主决定观看内容、时机与方式，实现查询驱动的高效视频理解。为训练此类智能体，我们设计了简单有效的三阶段学习流程——包括监督微调（SFT）、卡尼曼-特沃斯基优化（KTO）和广义奖励策略优化（GRPO）——从而桥接监督模仿与强化学习。我们还为各阶段构建高质量数据集，支持稳定可复现的训练。我们在六个视频理解基准测试中评估EVA，展示其综合能力。相比现有基线，EVA较通用MLLM基线提升6-12%，并较此前自适应智能体方法额外提升1-3%。代码与模型已公开。
URL
https://arxiv.org/abs/2603.22918
PDF
-
MVPBench: A Multi-Video Perception Evaluation Benchmark for Multi-Modal Video Understanding
2026-03-24 03:33:07Purui Bai, Tao Wu, Jiayang Sun, Xinyue Liu, Huaibo Huang, Ran HeAbstract
The rapid progress of Large Language Models (LLMs) has spurred growing interest in Multi-modal LLMs (MLLMs) and motivated the development of benchmarks to evaluate their perceptual and comprehension abilities. Existing benchmarks, however, are limited to static images or single videos, overlooking the complex interactions across multiple videos. To address this gap, we introduce the Multi-Video Perception Evaluation Benchmark (MVPBench), a new benchmark featuring 14 subtasks across diverse visual domains designed to evaluate models on extracting relevant information from video sequences to make informed decisions. MVPBench includes 5K question-answering tests involving 2.7K video clips sourced from existing datasets and manually annotated clips. Extensive evaluations reveal that current models struggle to process multi-video inputs effectively, underscoring substantial limitations in their multi-video comprehension. We anticipate MVPBench will drive advancements in multi-video perception.
Abstract (translated)
大语言模型（LLMs）的快速发展激发了人们对多模态大模型（MLLMs）日益浓厚的兴趣，并推动了用于评估其感知与理解能力的基准测试的开发。然而，现有基准测试局限于静态图像或单一视频，忽略了跨多个视频的复杂交互。为弥补这一空白，我们推出了多视频感知评估基准（MVPBench），这是一个涵盖多样视觉领域14个子任务的新基准，旨在评估模型从视频序列中提取相关信息以做出明智决策的能力。MVPBench包含5K个问答测试，涉及2.7K个来自现有数据集及手动标注的视频片段。广泛评估表明，当前模型难以有效处理多视频输入，凸显了其多视频理解能力的重大局限。我们期待MVPBench能推动多视频感知领域的发展。
URL
https://arxiv.org/abs/2603.22756
PDF
