Video_Caption
-
TrajTok: Learning Trajectory Tokens enables better Video Understanding
2026-02-26 09:15:34Chenhao Zheng, Jieyu Zhang, Jianing Zhang, Weikai Huang, Ashutosh Kumar, Quan Kong, Oncel Tuzel, Chun-Liang Li, Ranjay KrishnaAbstract
Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity, independent of video duration. TrajTok contains a unified segmenter that performs implicit clustering over pixels in both space and time to directly produce object trajectories in a single forward pass. By prioritizing downstream adaptability over pixel-perfect segmentation fidelity, TrajTok is lightweight and efficient, yet empirically improves video understanding performance. With TrajTok, we implement a video CLIP model trained from scratch (TrajViT2). It achieves the best accuracy at scale across both classification and retrieval benchmarks, while maintaining efficiency comparable to the best token-merging methods. TrajTok also proves to be a versatile component beyond its role as a tokenizer. We show that it can be seamlessly integrated as either a probing head for pretrained visual features (TrajAdapter) or an alignment connector in vision-language models (TrajVLM) with especially strong performance in long-video reasoning.
Abstract (translated)
视频模型中的 Tokenization 通常通过补丁化（patchification）实现，这会产生大量冗余的 token。这极大地限制了视频处理的效率和可扩展性。尽管最近基于轨迹的 tokenizer 提供了一种有前景的解决方案，能够将视频时长与 token 数量解耦，但它们依赖于复杂的外部分割和跟踪管道，这些管道运行缓慢且不针对特定任务。我们提出了 TrajTok，这是一种端到端的视频 tokenizer 模块，它完全集成并协同训练视频模型以适应下游目标，并根据语义复杂性动态调整其 token 粒度，不受视频时长的影响。 TrajTok 包含一个统一的分段器，它可以对空间和时间上的像素进行隐式聚类，直接在一个前向传递中生成对象轨迹。通过优先考虑下游任务适应性而不是像素级分割精度，TrajTok 既轻量又高效，并且在实践中提高了视频理解性能。 借助 TrajTok，我们实现了一个从头开始训练的视频 CLIP 模型（TrajViT2），它在分类和检索基准测试中实现了最佳的准确率扩展性，同时保持了与最好的 token 合并方法相当的效率。此外，TrajTok 也证明了其作为 tokenizer 超出角色之外的功能多样性。我们展示了它可以无缝地集成到预训练视觉特征的探测头（TrajAdapter）或视觉-语言模型中的对齐连接器（TrajVLM），尤其在长视频推理任务中表现出色。
URL
https://arxiv.org/abs/2602.22779
PDF
-
UniVBench: Towards Unified Evaluation for Video Foundation Models
2026-02-25 12:08:53Jianhui Wei, Xiaotian Zhang, Yichen Li, Yuan Wang, Yan Zhang, Ziyi Chen, Zhihang Tang, Wei Xu, Zuozhu LiuAbstract
Video foundation models aim to integrate video understanding, generation, editing, and instruction following within a single framework, making them a central direction for next-generation multimodal systems. However, existing evaluation benchmarks remain fragmented and limited in scope, as they each target a single task, rely on task-specific metrics, and typically use short or simple video clips. As a result, they do not capture the unified capabilities that these models are designed to deliver. To address this gap, we introduce UniVBench, a benchmark purpose-built for evaluating video foundation models across four core abilities: video understanding, video generation, video editing, and a newly proposed task, video reconstruction, which assesses how faithfully a model can reproduce video content it has encountered. Our benchmark substantially expands the complexity of evaluation by incorporating 200 high-quality, diverse and multi-shot videos, each paired with detailed captions, multi-format editing instructions, and reference images. All videos are human-created and carefully validated, offering richer cinematic information than prior benchmarks. In addition, we develop a unified agentic evaluation system (UniV-Eval) that standardizes prompting, instruction parsing, and scoring across all tasks, enabling fair, scalable, and reproducible comparisons of unified video models. By grounding evaluation in instruction-based multi-shot video tasks, UniVBench provides the first framework for measuring the integrated capabilities that video foundation models aim to achieve. Extensive human annotations ensure our evaluation aligns with human judgment, enabling rigorous assessment and accelerating progress toward robust video intelligence.
Abstract (translated)
视频基础模型的目标是将视频理解、生成、编辑和指令执行整合到一个单一的框架中，使之成为下一代多模态系统的核心发展方向。然而，现有的评估基准仍然分散且范围有限，因为它们各自针对单个任务，依赖特定于该任务的指标，并通常使用短小或简单的视频片段。因此，这些基准无法捕捉模型设计所要实现的统一能力。 为了填补这一空白，我们引入了UniVBench，这是一个专门用于评价视频基础模型在四大核心能力方面的基准：视频理解、视频生成、视频编辑和一项新提出的任务——视频重建，这项任务评估的是模型再现其遇到过的视频内容的忠实度。我们的基准通过纳入200部高质量、多样化且包含多镜头的视频来大幅扩展了评估的复杂性，每部视频都配有一系列详细的字幕、多种格式的编辑指令和参考图像。所有视频均由人类创作并经过精心验证，提供了比以往基准更加丰富的电影信息。 此外，我们开发了一个统一代理评估系统（UniV-Eval），标准化了所有任务中的提示生成、指令解析和评分过程，从而实现了公平、可扩展且可重复的统一视频模型比较。通过基于指令多镜头视频任务进行评价，UniVBench首次提供了一种测量视频基础模型所追求的整体能力的方法。 广泛的人员注释确保我们的评估与人类判断一致，使严格的评估成为可能，并加快向稳健视频智能发展的步伐。
URL
https://arxiv.org/abs/2602.21835
PDF
-
LongVideo-R1: Smart Navigation for Low-cost Long Video Understanding
2026-02-24 13:49:47Jihao Qiu, Lingxi Xie, Xinyue Huo, Qi Tian, Qixiang YeAbstract
This paper addresses the critical and underexplored challenge of long video understanding with low computational budgets. We propose LongVideo-R1, an active, reasoning-equipped multimodal large language model (MLLM) agent designed for efficient video context navigation, avoiding the redundancy of exhaustive search. At the core of LongVideo-R1 lies a reasoning module that leverages high-level visual cues to infer the most informative video clip for subsequent processing. During inference, the agent initiates traversal from top-level visual summaries and iteratively refines its focus, immediately halting the exploration process upon acquiring sufficient knowledge to answer the query. To facilitate training, we first extract hierarchical video captions from CGBench, a video corpus with grounding annotations, and guide GPT-5 to generate 33K high-quality chain-of-thought-with-tool trajectories. The LongVideo-R1 agent is fine-tuned upon the Qwen-3-8B model through a two-stage paradigm: supervised fine-tuning (SFT) followed by reinforcement learning (RL), where RL employs a specifically designed reward function to maximize selective and efficient clip navigation. Experiments on multiple long video benchmarks validate the effectiveness of name, which enjoys superior tradeoff between QA accuracy and efficiency. All curated data and source code are provided in the supplementary material and will be made publicly available. Code and data are available at: this https URL
Abstract (translated)
这篇论文解决了低计算预算下长视频理解这一关键且未充分探索的挑战。我们提出了LongVideo-R1，这是一种装备有推理能力的多模态大型语言模型（MLLM）代理，旨在实现高效的视频上下文导航，并避免详尽搜索带来的冗余问题。LongVideo-R1的核心在于一个推理模块，该模块利用高层次视觉线索来推断后续处理中最具信息量的视频片段。在进行推理时，代理从顶级视觉概览开始遍历，迭代地聚焦其关注点，并在获取足够知识以回答查询后立即停止探索过程。 为了方便训练，我们首先从CGBench（一个带有定位标注的视频语料库）中提取分层视频字幕，并指导GPT-5生成33K条高质量的链式思维与工具轨迹。LongVideo-R1代理通过两阶段范式在Qwen-3-8B模型上进行微调：第一阶段为监督微调（SFT），随后是强化学习（RL）。其中，RL采用专门设计的奖励函数来最大化选择性和高效的片段导航。实验表明，在多个长视频基准测试中，该方法在准确性和效率之间享有优越的权衡。所有整理好的数据和源代码都包含在补充材料中，并将公开发布。 代码和数据可在以下链接获得：this https URL
URL
https://arxiv.org/abs/2602.20913
PDF
-
PyVision-RL: Forging Open Agentic Vision Models via RL
2026-02-24 10:08:33Shitian Zhao, Shaoheng Lin, Ming Li, Haoquan Zhang, Wenshuo Peng, Kaipeng Zhang, Chen WeiAbstract
Reinforcement learning for agentic multimodal models often suffers from interaction collapse, where models learn to reduce tool usage and multi-turn reasoning, limiting the benefits of agentic behavior. We introduce PyVision-RL, a reinforcement learning framework for open-weight multimodal models that stabilizes training and sustains interaction. Our approach combines an oversampling-filtering-ranking rollout strategy with an accumulative tool reward to prevent collapse and encourage multi-turn tool use. Using a unified training pipeline, we develop PyVision-Image and PyVision-Video for image and video understanding. For video reasoning, PyVision-Video employs on-demand context construction, selectively sampling task-relevant frames during reasoning to significantly reduce visual token usage. Experiments show strong performance and improved efficiency, demonstrating that sustained interaction and on-demand visual processing are critical for scalable multimodal agents.
Abstract (translated)
基于代理的多模态模型在强化学习过程中常常会遇到交互崩溃的问题，即模型学会减少工具使用和多轮推理，从而限制了代理行为的好处。为此，我们引入了一个名为PyVision-RL的框架，它是一个面向开放权重多模态模型的强化学习框架，能够稳定训练并维持互动过程。 我们的方法结合了一种过采样-过滤-排序展开策略（oversampling-filtering-ranking rollout strategy）和累积工具奖励机制，以防止交互崩溃，并鼓励使用多轮工具。通过统一的训练流水线，我们开发了PyVision-Image和PyVision-Video，分别用于图像理解和视频理解。 对于视频推理，PyVision-Video采用按需构建上下文的方法，在推理过程中选择性地采样与任务相关的帧，从而大幅减少视觉标记使用量。实验结果表明，我们的方法在性能上表现出色，并提高了效率。这证明了可持续互动和按需视觉处理对可扩展多模态代理来说至关重要。
URL
https://arxiv.org/abs/2602.20739
PDF
-
QuickGrasp: Responsive Video-Language Querying Service via Accelerated Tokenization and Edge-Augmented Inference
2026-02-23 10:14:03Miao Zhang, Ruixiao Zhang, Jianxin Shi, Hengzhi Wang, Hao Fang, Jiangchuan LiuAbstract
Video-language models (VLMs) are reshaping video querying services, bringing unified solutions to complex perception and reasoning tasks. However, deploying large VLMs in real-world systems remains challenging due to their high resource demands, and remote-based deployment often results in unacceptable response delays. Although small, locally deployable VLMs offer faster responses, they unavoidably fall short in accuracy. To reconcile this trade-off, we propose QuickGrasp, a responsive, quality of service (QoS)-aware system that bridges this gap through a local-first architecture with on-demand edge augmentation. Built upon the highly modular architecture of VLMs, QuickGrasp shares the vision representation across model variants to avoid redundant computation. To maximize system-wide efficiency, QuickGrasp introduces three key designs: accelerated video tokenization, query-adaptive edge augmentation, and delay-aware, accuracy-preserving vision token density configuration. We implement a prototype of QuickGrasp and evaluate it across multiple video understanding benchmarks. The results show that QuickGrasp matches the accuracy of large VLMs while achieving up to a 12.8x reduction in response delay. QuickGrasp represents a key advancement toward building responsive video querying services for open-world understanding that fully leverage the capabilities of VLMs.
Abstract (translated)
视频语言模型（VLM）正在重塑视频查询服务，为复杂的感知和推理任务提供了统一的解决方案。然而，由于大型VLM对资源需求较高，在实际系统中部署它们仍然面临挑战，并且基于远程部署通常会导致不可接受的响应延迟。虽然小型、本地可部署的VLM可以提供更快的响应速度，但它们不可避免地在准确性方面表现较差。为了调和这一权衡，我们提出了QuickGrasp，这是一种响应迅速的服务质量（QoS）感知系统，通过一种以本地为主导架构并按需边缘增强的方式来弥合这一差距。基于VLM高度模块化的架构，QuickGrasp共享不同模型变体间的视觉表示，避免了重复计算。为了最大化整个系统的效率，QuickGrasp引入了三个关键设计：加速视频标记化、查询自适应边缘增强以及延迟感知和准确性保持的视觉令牌密度配置。 我们实施了一个QuickGrasp原型，并在多个视频理解基准上对其进行了评估。结果显示，QuickGrasp能够在与大型VLM匹敌准确度的同时，将响应延迟最多降低12.8倍。QuickGrasp代表了构建响应迅速的视频查询服务的关键进展，这些服务能够充分运用VLM的能力来应对开放世界理解中的挑战。
URL
https://arxiv.org/abs/2603.00126
PDF
-
Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation
2026-02-22 13:26:27Thanh Nguyen Canh, Thanh-Tuan Tran, Haolan Zhang, Ziyan Gao, Nak Young Chong, Xiem HoangVanAbstract
Learning from Demonstration (LfD) offers a promising paradigm for robot skill acquisition. Recent approaches attempt to extract manipulation commands directly from video demonstrations, yet face two critical challenges: (1) general video captioning models prioritize global scene features over task-relevant objects, producing descriptions unsuitable for precise robotic execution, and (2) end-to-end architectures coupling visual understanding with policy learning require extensive paired datasets and struggle to generalize across objects and scenarios. To address these limitations, we propose a novel ``Human-to-Robot'' imitation learning pipeline that enables robots to acquire manipulation skills directly from unstructured video demonstrations, inspired by the human ability to learn by watching and imitating. Our key innovation is a modular framework that decouples the learning process into two distinct stages: (1) Video Understanding, which combines Temporal Shift Modules (TSM) with Vision-Language Models (VLMs) to extract actions and identify interacted objects, and (2) Robot Imitation, which employs TD3-based deep reinforcement learning to execute the demonstrated manipulations. We validated our approach in PyBullet simulation environments with a UR5e manipulator and in a real-world experiment with a UF850 manipulator across four fundamental actions: reach, pick, move, and put. For video understanding, our method achieves 89.97% action classification accuracy and BLEU-4 scores of 0.351 on standard objects and 0.265 on novel objects, representing improvements of 76.4% and 128.4% over the best baseline, respectively. For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations. The project website is available at this https URL.
Abstract (translated)
从演示学习（LfD）为机器人技能获取提供了一种有前景的范式。最近的方法试图直接从视频演示中提取操作命令，但面临着两个关键挑战：(1) 通用视频描述模型优先考虑全局场景特征而非与任务相关的对象，从而生成不适用于精确机器人执行的描述；(2) 将视觉理解与策略学习结合在一起的端到端架构需要大量的成对数据集，并且难以在不同物体和情景中泛化。为了解决这些局限性，我们提出了一种新型“人机”模仿学习流水线，使机器人能够直接从无结构视频演示中获取操作技能，灵感来自于人类通过观察和模仿来学习的能力。我们的关键创新是一个模块化的框架，将学习过程分解成两个独立的阶段：(1) 视频理解，结合时间偏移模块（TSM）与视觉-语言模型（VLM），以提取动作并识别交互对象；(2) 机器人模仿，使用基于TD3的深度强化学习来执行演示的操作。我们在PyBullet模拟环境中使用UR5e机械臂以及在现实世界实验中使用UF850机械臂进行了验证，涵盖了四个基本操作：到达、抓取、移动和放置。 对于视频理解，我们的方法达到了89.97%的动作分类准确率，并在标准对象上获得了BLEU-4得分为0.351，在新对象上的得分是0.265，分别比最佳基线提高了76.4% 和 128.4%。对于机器人操作，我们的框架在所有动作中达到了平均成功率87.5%，到达任务的成功率为100%，而复杂的抓放操作可达90%的最高成功率。 项目网站可在以下链接访问：[此处插入实际网址]（原文中的“this https URL”应替换为具体网址）。
URL
https://arxiv.org/abs/2602.19184
PDF
-
Frame2Freq: Spectral Adapters for Fine-Grained Video Understanding
2026-02-21 23:05:53Thinesh Thiyakesan Ponbagavathi, Constantin Seibold, Alina RoitbergAbstract
Adapting image-pretrained backbones to video typically relies on time-domain adapters tuned to a single temporal scale. Our experiments show that these modules pick up static image cues and very fast flicker changes, while overlooking medium-speed motion. Capturing dynamics across multiple time-scales is, however, crucial for fine-grained temporal analysis (i.e., opening vs. closing bottle). To address this, we introduce Frame2Freq -- a family of frequency-aware adapters that perform spectral encoding during image-to-video adaptation of pretrained Vision Foundation Models (VFMs), improving fine-grained action recognition. Frame2Freq uses Fast Fourier Transform (FFT) along time and learns frequency-band specific embeddings that adaptively highlight the most discriminative frequency ranges. Across five fine-grained activity recognition datasets, Frame2Freq outperforms prior PEFT methods and even surpasses fully fine-tuned models on four of them. These results provide encouraging evidence that frequency analysis methods are a powerful tool for modeling temporal dynamics in image-to-video transfer. Code is available at this https URL.
Abstract (translated)
将图像预训练骨干网络适应到视频中通常依赖于针对单一时间尺度调优的时间域适配器。我们的实验表明，这些模块倾向于捕捉静态图像线索和非常快速的闪烁变化，而忽略了中速运动。然而，跨多个时间尺度捕获动态对于细粒度的时间分析（例如，开瓶与关瓶）至关重要。为了解决这一问题，我们引入了Frame2Freq——一种频率感知适配器家族，在将预训练的视觉基础模型（VFMs）从图像转换到视频时进行频谱编码，从而改进细粒度的动作识别。 Frame2Freq使用快速傅里叶变换（FFT）沿时间轴执行，并学习特定于频率带的嵌入以自适应地突出最具判别性的频率范围。在五个细粒度活动识别数据集中，与以前的PEFT方法相比，Frame2Freq表现出色，在其中四个上甚至超过了完全微调的模型。这些结果提供了令人鼓舞的证据，表明频谱分析方法是建模图像到视频转换中的时间动态的强大工具。 代码可在以下链接获取：[此 URL]（请将此处的URL替换为实际提供的GitHub或其他存储库链接）。
URL
https://arxiv.org/abs/2602.18977
PDF
-
Think with Grounding: Curriculum Reinforced Reasoning with Video Grounding for Long Video Understanding
2026-02-21 03:16:23Houlun Chen, Xin Wang, Guangyao Li, Yuwei Zhou, Yihan Chen, Jia Jia, Wenwu ZhuAbstract
Long video understanding is challenging due to rich and complicated multimodal clues in long temporal this http URL methods adopt reasoning to improve the model's ability to analyze complex video clues in long videos via text-form this http URL,the existing literature suffers from the fact that the text-only reasoning under fixed video context may exacerbate hallucinations since detailed crucial clues are often ignored under limited video context length due to the temporal redundancy of long this http URL address this gap,we propose Video-TwG,a curriculum reinforced framework that employs a novel Think-with-Grounding paradigm,enabling video LLMs to actively decide when to perform on-demand grounding during interleaved text-video reasoning, selectively zooming into question-relevant clips only when this http URL-TwG can be trained end-to-end in a straightforward manner, without relying on complex auxiliary modules or heavily annotated reasoning tracesIn detail,we design a Two-stage Reinforced Curriculum Strategy, where the model first learns think-with-grounding behavior on a small short-video GQA dataset with grounding labels,and then scales to diverse general QA data with videos of diverse domains to encourage generalization. Further, to handle complex think-with-grounding reasoning for various kinds of data,we propose TwG-GRPO algorithm which features the fine-grained grounding reward, self-confirmed pseudo reward and accuracy-gated this http URL,we propose to construct a new TwG-51K dataset that facilitates training. Experiments on Video-MME, LongVideoBench, and MLVU show that Video-TwG consistently outperforms strong LVU this http URL ablation validates the necessity of our Two-stage Reinforced Curriculum Strategy and shows our TwG-GRPO better leverages diverse unlabeled data to improve grounding quality and reduce redundant groundings without sacrificing QA performance.
Abstract (translated)
长视频理解是一项挑战，因为其中包含丰富且复杂的多模态线索。现有方法通过采用推理来增强模型分析长时间视频复杂线索的能力，并将这种能力表示为文本形式的逻辑表达。然而，现有的文献存在一个问题：仅依赖固定视频上下文进行文本推理可能会加剧幻觉现象，因为在有限的视频上下文中，往往忽略了一些详细的、关键性的线索，而这些线索在冗长的视频中是至关重要的。 为了填补这一空白，我们提出了Video-TwG（Think-with-Grounding 视频框架），这是一个采用新型思考与接地范式的课程强化框架。该框架使视频LLM能够在交错的文字-视频推理过程中主动决定何时进行按需接地操作，并在仅针对问题相关片段时执行此操作。 Video-TwG可以以端到端的简便方式训练，无需依赖复杂的辅助模块或密集标注的推理轨迹。具体而言，我们设计了一种两阶段强化课程策略：模型首先在一个小规模的短视频问答数据集上学习思考与接地行为，并在此过程中使用了接地标签；随后扩展至涵盖不同领域多种类型的通用问答数据，以鼓励泛化能力。 为应对各种类型数据复杂的思考-接地推理问题，我们提出了TwG-GRPO算法，该算法具有细粒度的接地奖励、自我确认伪奖励和准确率门控机制的特点。此外，为了便于训练，我们提议构建一个新的TwG-51K数据集。 在Video-MME, LongVideoBench 和 MLVU上的实验表明，Video-TwG在长视频理解方面持续超越强大的LVU方法。消融分析验证了我们的两阶段强化课程策略的必要性，并显示了我们的TwG-GRPO算法能够更好地利用多样化未标记数据来提高接地质量并减少冗余的接地操作，同时不牺牲问答性能。 这种方法通过一种创新的方式解决了长视频理解中的核心问题，即如何在复杂的多模态环境中有效地提取和处理关键信息。
URL
https://arxiv.org/abs/2602.18702
PDF
-
Going Down Memory Lane: Scaling Tokens for Video Stream Understanding with Dynamic KV-Cache Memory
2026-02-20 18:59:50Vatsal Agarwal, Saksham Suri, Matthew Gwilliam, Pulkit Kumar, Abhinav ShrivastavaAbstract
Streaming video understanding requires models to robustly encode, store, and retrieve information from a continuous video stream to support accurate video question answering (VQA). Existing state-of-the-art approaches rely on key-value caching to accumulate frame-level information over time, but use a limited number of tokens per frame, leading to the loss of fine-grained visual details. In this work, we propose scaling the token budget to enable more granular spatiotemporal understanding and reasoning. First, we find that current methods are ill-equipped to handle dense streams: their feature encoding causes query-frame similarity scores to increase over time, biasing retrieval toward later frames. To address this, we introduce an adaptive selection strategy that reduces token redundancy while preserving local spatiotemporal information. We further propose a training-free retrieval mixture-of-experts that leverages external models to better identify relevant frames. Our method, MemStream, achieves +8.0% on CG-Bench, +8.5% on LVBench, and +2.4% on VideoMME (Long) over ReKV with Qwen2.5-VL-7B.
Abstract (translated)
流媒体视频理解需要模型能够稳健地编码、存储和从连续的视频流中检索信息，以支持准确的视频问答（VQA）。现有的最先进的方法依赖于键值缓存来随着时间积累帧级信息，但每帧使用的令牌数量有限，导致丢失了细微的视觉细节。在本工作中，我们提出增加令牌预算，以便实现更细致的空间和时间理解与推理。首先，我们发现当前的方法处理密集流时存在不足：它们的特征编码会导致查询-帧相似度分数随时间增加，偏向于检索较晚的帧。为解决这一问题，我们引入了一种自适应选择策略，在减少令牌冗余的同时保留局部时空信息。此外，我们提出了一种无需训练的混合专家检索方法，利用外部模型更好地识别相关帧。我们的方法MemStream在CG-Bench上比ReKV with Qwen2.5-VL-7B提高了8.0%，在LVBench上提高了8.5%，在VideoMME（Long）上提高了2.4%。
URL
https://arxiv.org/abs/2602.18434
PDF
-
DeepSVU: Towards In-depth Security-oriented Video Understanding via Unified Physical-world Regularized MoE
2026-02-20 06:18:32Yujie Jin, Wenxin Zhang, Jingjing Wang, Guodong ZhouAbstract
In the literature, prior research on Security-oriented Video Understanding (SVU) has predominantly focused on detecting and localize the threats (e.g., shootings, robberies) in videos, while largely lacking the effective capability to generate and evaluate the threat causes. Motivated by these gaps, this paper introduces a new chat paradigm SVU task, i.e., In-depth Security-oriented Video Understanding (DeepSVU), which aims to not only identify and locate the threats but also attribute and evaluate the causes threatening segments. Furthermore, this paper reveals two key challenges in the proposed task: 1) how to effectively model the coarse-to-fine physical-world information (e.g., human behavior, object interactions and background context) to boost the DeepSVU task; and 2) how to adaptively trade off these factors. To tackle these challenges, this paper proposes a new Unified Physical-world Regularized MoE (UPRM) approach. Specifically, UPRM incorporates two key components: the Unified Physical-world Enhanced MoE (UPE) Block and the Physical-world Trade-off Regularizer (PTR), to address the above two challenges, respectively. Extensive experiments conduct on our DeepSVU instructions datasets (i.e., UCF-C instructions and CUVA instructions) demonstrate that UPRM outperforms several advanced Video-LLMs as well as non-VLM approaches. Such this http URL justify the importance of the coarse-to-fine physical-world information in the DeepSVU task and demonstrate the effectiveness of our UPRM in capturing such information.
Abstract (translated)
在相关文献中，面向安全的视频理解（Security-oriented Video Understanding, SVU）的研究主要集中在检测和定位视频中的威胁行为（如枪击、抢劫等），但较少关注生成和评估这些威胁的原因。鉴于这些研究空白，本文提出了一种新的基于对话的SVU任务——深入的安全视频理解（In-depth Security-oriented Video Understanding，DeepSVU）。这一新任务旨在不仅识别并定位威胁，还对威胁原因进行归因和评价。 为了应对这一新任务中的挑战，本文指出两个关键问题： 1. 如何有效地建模从粗到细的物理世界信息（例如人类行为、物体交互以及背景环境），以促进DeepSVU任务。 2. 如何适应性地权衡这些因素之间的关系。 为了解决上述挑战，本文提出了一种新的统一物理世界正则化Mixture-of-Experts (MoE)方法——UPRM。具体来说，UPRM包含了两个核心组件：统一物理世界增强的MoE（Unified Physical-world Enhanced MoE, UPE）模块和物理世界的权衡正则器（Physical-world Trade-off Regularizer, PTR），分别解决上述两大挑战。 在我们构建的DeepSVU指令数据集（即UCF-C指令和CUVA指令）上进行了广泛的实验，结果表明UPRM优于几个先进的视频LLMs以及非VLM方法。这样的实验结果证明了粗到细物理世界信息在DeepSVU任务中的重要性，并展示了我们的UPRM在捕捉这些信息方面的有效性。 请注意，在最后的句子中，“Such this http URL” 应该是一个链接或者引用，但由于原文可能有错误或排版问题，这里并未提供具体网址。如果这是一个有意为之的参考，请根据上下文调整其表述以符合语言习惯和语义连贯性。
URL
https://arxiv.org/abs/2602.18019
PDF
-
Learning Compact Video Representations for Efficient Long-form Video Understanding in Large Multimodal Models
2026-02-19 22:04:27Yuxiao Chen, Jue Wang, Zhikang Zhang, Jingru Yi, Xu Zhang, Yang Zou, Zhaowei Cai, Jianbo Yuan, Xinyu Li, Hao Yang, Davide ModoloAbstract
With recent advancements in video backbone architectures, combined with the remarkable achievements of large language models (LLMs), the analysis of long-form videos spanning tens of minutes has become both feasible and increasingly prevalent. However, the inherently redundant nature of video sequences poses significant challenges for contemporary state-of-the-art models. These challenges stem from two primary aspects: 1) efficiently incorporating a larger number of frames within memory constraints, and 2) extracting discriminative information from the vast volume of input data. In this paper, we introduce a novel end-to-end schema for long-form video understanding, which includes an information-density-based adaptive video sampler (AVS) and an autoencoder-based spatiotemporal video compressor (SVC) integrated with a multimodal large language model (MLLM). Our proposed system offers two major advantages: it adaptively and effectively captures essential information from video sequences of varying durations, and it achieves high compression rates while preserving crucial discriminative information. The proposed framework demonstrates promising performance across various benchmarks, excelling in both long-form video understanding tasks and standard video understanding benchmarks. These results underscore the versatility and efficacy of our approach, particularly in managing the complexities of prolonged video sequences.
Abstract (translated)
随着视频骨干网络架构的最新进展以及大型语言模型（LLM）的显著成就，对长达数分钟的长视频进行分析变得既可行又日益普遍。然而，视频序列本身固有的冗余性为现代最先进的模型带来了重大挑战。这些挑战主要源自两个方面：1) 在内存限制下有效整合更多帧，2) 从大量的输入数据中提取判别信息。 在本文中，我们提出了一种新的端到端长视频理解方案，包括基于信息密度的自适应视频采样器（AVS）和与多模态大型语言模型（MLLM）集成的空间时间视频压缩器（SVC）。我们的系统具有两个主要优势：它能够根据不同长度的视频序列自适应且有效地捕捉重要信息，并在保持关键判别性信息的同时实现高压缩率。所提出的框架在各种基准测试中表现出色，尤其擅长长视频理解任务和标准视频理解基准。 这些结果强调了我们方法的灵活性和有效性，特别是在处理长期视频序列复杂性的方面。
URL
https://arxiv.org/abs/2602.17869
PDF
-
KPM-Bench: A Kinematic Parsing Motion Benchmark for Fine-grained Motion-centric Video Understanding
2026-02-19 19:01:09Boda Lin, Yongjie Zhu, Xiaocheng Gong, Wenyu Qin, Meng WangAbstract
Despite recent advancements, video captioning models still face significant limitations in accurately describing fine-grained motion details and suffer from severe hallucination issues. These challenges become particularly prominent when generating captions for motion-centric videos, where precise depiction of intricate movements and limb dynamics is crucial yet often neglected. To alleviate this gap, we introduce an automated annotation pipeline that integrates kinematic-based motion computation with linguistic parsing, enabling detailed decomposition and description of complex human motions. Based on this pipeline, we construct and release the Kinematic Parsing Motion Benchmark (KPM-Bench), a novel open-source dataset designed to facilitate fine-grained motion understanding. KPM-Bench consists of (i) fine-grained video-caption pairs that comprehensively illustrate limb-level dynamics in complex actions, (ii) diverse and challenging question-answer pairs focusing specifically on motion understanding, and (iii) a meticulously curated evaluation set specifically designed to assess hallucination phenomena associated with motion descriptions. Furthermore, to address hallucination issues systematically, we propose the linguistically grounded Motion Parsing and Extraction (MoPE) algorithm, capable of accurately extracting motion-specific attributes directly from textual captions. Leveraging MoPE, we introduce a precise hallucination evaluation metric that functions independently of large-scale vision-language or language-only models. By integrating MoPE into the GRPO post-training framework, we effectively mitigate hallucination problems, significantly improving the reliability of motion-centric video captioning models.
Abstract (translated)
尽管近期取得了进展，视频字幕模型在准确描述精细的运动细节方面仍然面临重大局限，并且存在严重的幻觉问题。这些问题在生成以动作为中心的视频的字幕时尤为突出，在这些情况下，精确描绘复杂的肢体动态至关重要但往往被忽视。为了解决这一差距，我们引入了一种自动注释流水线，该流水线结合了基于运动学的运动计算和语言解析功能，能够详细地分解并描述复杂的人体动作。在此基础上，我们构建并发布了Kinematic Parsing Motion Benchmark（KPM-Bench），这是一个旨在促进精细运动理解的新颖开源数据集。 KPM-Bench 包括： (i) 精细粒度的视频-字幕对，全面展示在复杂的动作中肢体级别的动态变化； (ii) 一系列多样且具有挑战性的问答对，专门针对运动理解设计；以及 (iii) 一套精心策划的评估集，专门用于评估与运动描述相关的幻觉现象。 此外，为了系统地解决幻觉问题，我们提出了一种基于语言基础的Motion Parsing and Extraction（MoPE）算法，能够从文本字幕中准确提取特定于运动的属性。利用 MoPE，我们引入了一个精确的幻觉评估指标，该指标可以独立于大规模视觉-语言或仅语言模型运行。 通过将 MoPE 集成到 GRPO 后训练框架中，我们有效地减轻了幻觉问题，并显著提高了以动作为中心的视频字幕模型的可靠性。
URL
https://arxiv.org/abs/2602.17768
PDF
-
Xray-Visual Models: Scaling Vision models on Industry Scale Data
2026-02-18 22:22:44Shlok Mishra, Tsung-Yu Lin, Linda Wang, Hongli Xu, Yimin Liu, Michael Hsu, Chaitanya Ahuja, Hao Yuan, Jianpeng Cheng, Hong-You Chen, Haoyuan Xu, Chao Li, Abhijeet Awasthi, Jihye Moon, Don Husa, Michael Ge, Sumedha Singla, Arkabandhu Chowdhury, Phong Dingh, Satya Narayan Shukla, Yonghuan Yang, David Jacobs, Qi Guo, Jun Xiao, Xiangjun Fan, Aashu SinghAbstract
We present Xray-Visual, a unified vision model architecture for large-scale image and video understanding trained on industry-scale social media data. Our model leverages over 15 billion curated image-text pairs and 10 billion video-hashtag pairs from Facebook and Instagram, employing robust data curation pipelines that incorporate balancing and noise suppression strategies to maximize semantic diversity while minimizing label noise. We introduce a three-stage training pipeline that combines self-supervised MAE, semi-supervised hashtag classification, and CLIP-style contrastive learning to jointly optimize image and video modalities. Our architecture builds on a Vision Transformer backbone enhanced with efficient token reorganization (EViT) for improved computational efficiency. Extensive experiments demonstrate that Xray-Visual achieves state-of-the-art performance across diverse benchmarks, including ImageNet for image classification, Kinetics and HMDB51 for video understanding, and MSCOCO for cross-modal retrieval. The model exhibits strong robustness to domain shift and adversarial perturbations. We further demonstrate that integrating large language models as text encoders (LLM2CLIP) significantly enhances retrieval performance and generalization capabilities, particularly in real-world environments. Xray-Visual establishes new benchmarks for scalable, multimodal vision models, while maintaining superior accuracy and computational efficiency.
Abstract (translated)
URL
https://arxiv.org/abs/2602.16918
PDF
-
Let's Split Up: Zero-Shot Classifier Edits for Fine-Grained Video Understanding
2026-02-18 15:46:36Kaiting Liu, Hazel DoughtyAbstract
Video recognition models are typically trained on fixed taxonomies which are often too coarse, collapsing distinctions in object, manner or outcome under a single label. As tasks and definitions evolve, such models cannot accommodate emerging distinctions and collecting new annotations and retraining to accommodate such changes is costly. To address these challenges, we introduce category splitting, a new task where an existing classifier is edited to refine a coarse category into finer subcategories, while preserving accuracy elsewhere. We propose a zero-shot editing method that leverages the latent compositional structure of video classifiers to expose fine-grained distinctions without additional data. We further show that low-shot fine-tuning, while simple, is highly effective and benefits from our zero-shot initialization. Experiments on our new video benchmarks for category splitting demonstrate that our method substantially outperforms vision-language baselines, improving accuracy on the newly split categories without sacrificing performance on the rest. Project page: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.16545
PDF
-
ReMoRa: Multimodal Large Language Model based on Refined Motion Representation for Long-Video Understanding
2026-02-18 12:37:35Daichi Yashima, Shuhei Kurita, Yusuke Oda, Komei SugiuraAbstract
While multimodal large language models (MLLMs) have shown remarkable success across a wide range of tasks, long-form video understanding remains a significant challenge. In this study, we focus on video understanding by MLLMs. This task is challenging because processing a full stream of RGB frames is computationally intractable and highly redundant, as self-attention have quadratic complexity with sequence length. In this paper, we propose ReMoRa, a video MLLM that processes videos by operating directly on their compressed representations. A sparse set of RGB keyframes is retained for appearance, while temporal dynamics are encoded as a motion representation, removing the need for sequential RGB frames. These motion representations act as a compact proxy for optical flow, capturing temporal dynamics without full frame decoding. To refine the noise and low fidelity of block-based motions, we introduce a module to denoise and generate a fine-grained motion representation. Furthermore, our model compresses these features in a way that scales linearly with sequence length. We demonstrate the effectiveness of ReMoRa through extensive experiments across a comprehensive suite of long-video understanding benchmarks. ReMoRa outperformed baseline methods on multiple challenging benchmarks, including LongVideoBench, NExT-QA, and MLVU.
Abstract (translated)
URL
https://arxiv.org/abs/2602.16412
PDF
-
DataCube: A Video Retrieval Platform via Natural Language Semantic Profiling
2026-02-18 07:12:01Yiming Ju, Hanyu Zhao, Quanyue Ma, Donglin Hao, Chengwei Wu, Ming Li, Songjing Wang, Tengfei PanAbstract
Large-scale video repositories are increasingly available for modern video understanding and generation tasks. However, transforming raw videos into high-quality, task-specific datasets remains costly and inefficient. We present DataCube, an intelligent platform for automatic video processing, multi-dimensional profiling, and query-driven retrieval. DataCube constructs structured semantic representations of video clips and supports hybrid retrieval with neural re-ranking and deep semantic matching. Through an interactive web interface, users can efficiently construct customized video subsets from massive repositories for training, analysis, and evaluation, and build searchable systems over their own private video collections. The system is publicly accessible at this https URL. Demo Video: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2602.16231
PDF
-
EventMemAgent: Hierarchical Event-Centric Memory for Online Video Understanding with Adaptive Tool Use
2026-02-17 03:17:24Siwei Wen, Zhangcheng Wang, Xingjian Zhang, Lei Huang, Wenjun WuAbstract
Online video understanding requires models to perform continuous perception and long-range reasoning within potentially infinite visual streams. Its fundamental challenge lies in the conflict between the unbounded nature of streaming media input and the limited context window of Multimodal Large Language Models (MLLMs). Current methods primarily rely on passive processing, which often face a trade-off between maintaining long-range context and capturing the fine-grained details necessary for complex tasks. To address this, we introduce EventMemAgent, an active online video agent framework based on a hierarchical memory module. Our framework employs a dual-layer strategy for online videos: short-term memory detects event boundaries and utilizes event-granular reservoir sampling to process streaming video frames within a fixed-length buffer dynamically; long-term memory structuredly archives past observations on an event-by-event basis. Furthermore, we integrate a multi-granular perception toolkit for active, iterative evidence capture and employ Agentic Reinforcement Learning (Agentic RL) to end-to-end internalize reasoning and tool-use strategies into the agent's intrinsic capabilities. Experiments show that EventMemAgent achieves competitive results on online video benchmarks. The code will be released here: this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.15329
PDF
-
Dual-Signal Adaptive KV-Cache Optimization for Long-Form Video Understanding in Vision-Language Models
2026-02-15 17:06:02Vishnu Sai, Dheeraj Sai, Srinath B, Girish Varma, Priyesh ShuklaAbstract
Vision-Language Models (VLMs) face a critical memory bottleneck when processing long-form video content due to the linear growth of the Key-Value (KV) cache with sequence length. Existing solutions predominantly employ reactive eviction strategies that compute full attention matrices before discarding tokens, resulting in substantial computational waste. We propose Sali-Cache, a novel a priori optimization framework that implements dual-signal adaptive caching through proactive memory management. By integrating a temporal filter based on optical flow analysis for detecting inter-frame redundancy and a spatial filter leveraging saliency detection for identifying visually significant regions, Sali-Cache intelligently manages memory allocation before entering computationally expensive attention operations. Experimental evaluation on the LLaVA 1.6 architecture demonstrates that our method achieves a 2.20x compression ratio in effective memory usage while maintaining 100% accuracy across BLEU, ROUGE-L, and Exact Match metrics. Furthermore, under identical memory budget constraints, Sali-Cache preserves context-rich features over extended temporal durations without degrading model performance, enabling efficient processing of long-form video content on consumer-grade hardware.
Abstract (translated)
URL
https://arxiv.org/abs/2602.14236
PDF
-
A generalizable foundation model for intraoperative understanding across surgical procedures
2026-02-14 06:52:42Kanggil Park, Yongjun Jeon, Soyoung Lim, Seonmin Park, Jongmin Shin, Jung Yong Kim, Sehyeon An, Jinsoo Rhu, Jongman Kim, Gyu-Seong Choi, Namkee Oh, Kyu-Hwan JungAbstract
In minimally invasive surgery, clinical decisions depend on real-time visual interpretation, yet intraoperative perception varies substantially across surgeons and procedures. This variability limits consistent assessment, training, and the development of reliable artificial intelligence systems, as most surgical AI models are designed for narrowly defined tasks and do not generalize across procedures or institutions. Here we introduce ZEN, a generalizable foundation model for intraoperative surgical video understanding trained on more than 4 million frames from over 21 procedures using a self-supervised multi-teacher distillation framework. We curated a large and diverse dataset and systematically evaluated multiple representation learning strategies within a unified benchmark. Across 20 downstream tasks and full fine-tuning, frozen-backbone, few-shot and zero-shot settings, ZEN consistently outperforms existing surgical foundation models and demonstrates robust cross-procedure generalization. These results suggest a step toward unified representations for surgical scene understanding and support future applications in intraoperative assistance and surgical training assessment.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13633
PDF
-
Towards Sparse Video Understanding and Reasoning
2026-02-14 04:52:11Chenwei Xu, Zhen Ye, Shang Wu, Weijian Li, Zihan Wang, Zhuofan Xia, Lie Lu, Pranav Maneriker, Fan Du, Manling Li, Han LiuAbstract
We present \revise (\underline{Re}asoning with \underline{Vi}deo \underline{S}parsity), a multi-round agent for video question answering (VQA). Instead of uniformly sampling frames, \revise selects a small set of informative frames, maintains a summary-as-state across rounds, and stops early when confident. It supports proprietary vision-language models (VLMs) in a ``plug-and-play'' setting and enables reinforcement fine-tuning for open-source models. For fine-tuning, we introduce EAGER (Evidence-Adjusted Gain for Efficient Reasoning), an annotation-free reward with three terms: (1) Confidence gain: after new frames are added, we reward the increase in the log-odds gap between the correct option and the strongest alternative; (2) Summary sufficiency: at answer time we re-ask using only the last committed summary and reward success; (3) Correct-and-early stop: answering correctly within a small turn budget is rewarded. Across multiple VQA benchmarks, \revise improves accuracy while reducing frames, rounds, and prompt tokens, demonstrating practical sparse video reasoning.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13602
PDF
