Video_Caption
-
TrajLoom: Dense Future Trajectory Generation from Video
2026-03-23 22:10:58Zewei Zhang, Jia Jun Cheng Xian, Kaiwen Liu, Ming Liang, Hang Chu, Jun Chen, Renjie LiaoAbstract
Predicting future motion is crucial in video understanding and controllable video generation. Dense point trajectories are a compact, expressive motion representation, but modeling their future evolution from observed video remains challenging. We propose a framework that predicts future trajectories and visibility from past trajectories and video context. Our method has three components: (1) Grid-Anchor Offset Encoding, which reduces location-dependent bias by representing each point as an offset from its pixel-center anchor; (2) TrajLoom-VAE, which learns a compact spatiotemporal latent space for dense trajectories with masked reconstruction and a spatiotemporal consistency regularizer; and (3) TrajLoom-Flow, which generates future trajectories in latent space via flow matching, with boundary cues and on-policy K-step fine-tuning for stable sampling. We also introduce TrajLoomBench, a unified benchmark spanning real and synthetic videos with a standardized setup aligned with video-generation benchmarks. Compared with state-of-the-art methods, our approach extends the prediction horizon from 24 to 81 frames while improving motion realism and stability across datasets. The predicted trajectories directly support downstream video generation and editing. Code, model checkpoints, and datasets are available at this https URL.
Abstract (translated)
预测未来运动在视频理解与可控视频生成中至关重要。密集点轨迹是一种紧凑且富有表现力的运动表征，但从观测视频中建模其未来演变仍具有挑战性。我们提出了一种从历史轨迹与视频上下文预测未来轨迹及可见性的框架。该方法包含三个核心组件：(1) 网格锚点偏移编码（Grid-Anchor Offset Encoding），通过将每个点表示为其像素中心锚点的偏移量，减轻位置依赖偏差；(2) TrajLoom-VAE，借助掩码重建与时空一致性正则化器，为密集轨迹学习紧凑的时空潜在空间；(3) TrajLoom-Flow，通过流匹配在潜在空间中生成未来轨迹，并利用边界线索与同策略K步微调实现稳定采样。我们还引入了TrajLoomBench，这是一个覆盖真实与合成视频的统一基准，其标准化设置与视频生成基准对齐。相较于最先进方法，我们的方案将预测范围从24帧扩展到81帧，同时在跨数据集上提升了运动真实性与稳定性。预测轨迹可直接支持下游视频生成与编辑任务。代码、模型检查点及数据集已公开于此链接。
URL
https://arxiv.org/abs/2603.22606
PDF
-
Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos
2026-03-23 19:47:58Shoubin Yu, Lei Shu, Antoine Yang, Yao Fu, Srinivas Sunkara, Maria Wang, Jindong Chen, Mohit Bansal, Boqing GongAbstract
Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limitation: they focus entirely on web-based interaction and perception, lacking grounding in the user's real-world physical surroundings. This limitation prevents evaluation in crucial scenarios, such as when an agent must use egocentric visual perception (e.g., via AR glasses) to recognize an object in the user's surroundings and then complete a related task online. To address this gap, we introduce Ego2Web, the first benchmark designed to bridge egocentric video perception and web agent execution. Ego2Web pairs real-world first-person video recordings with web tasks that require visual understanding, web task planning, and interaction in an online environment for successful completion. We utilize an automatic data-generation pipeline combined with human verification and refinement to curate well-constructed, high-quality video-task pairs across diverse web task types, including e-commerce, media retrieval, knowledge lookup, etc. To facilitate accurate and scalable evaluation for our benchmark, we also develop a novel LLM-as-a-Judge automatic evaluation method, Ego2WebJudge, which achieves approximately 84% agreement with human judgment, substantially higher than existing evaluation methods. Experiments with diverse SoTA agents on our Ego2Web show that their performance is weak, with substantial headroom across all task categories. We also conduct a comprehensive ablation study on task design, highlighting the necessity of accurate video understanding in the proposed task and the limitations of current agents. We hope Ego2Web can be a critical new resource for developing truly capable AI assistants that can seamlessly see, understand, and act across the physical and digital worlds.
Abstract (translated)
多模态AI智能体正日益自动化涉及在线网络执行的复杂现实工作流。然而，当前网络智能体基准存在一个关键局限：它们完全聚焦于基于网络的交互与感知，缺乏对用户真实物理环境的锚定。这一局限导致在关键场景中无法进行评估，例如当智能体必须通过以自我为中心的视觉感知（如通过AR眼镜）识别用户周围环境中的物体，然后在线完成相关任务。为弥补这一空白，我们推出了Ego2Web——首个旨在连接第一人称视频感知与网络智能体执行的基准测试。Ego2Web将真实世界的第一人称视频记录与需要视觉理解、网络任务规划及在线环境交互才能成功完成的任务进行配对。我们利用自动数据生成管道，结合人工验证与精炼， curation了涵盖电商、媒体检索、知识查询等多种网络任务类型的构造精良、高质量视频-任务对。为支持该基准的准确且可扩展评估，我们还开发了一种新颖的LLM-as-a-Judge自动评估方法Ego2WebJudge，其与人类判断的吻合度约84%，显著高于现有评估方法。在Ego2Web上使用多样化的SOTA智能体进行实验表明，其性能普遍较弱，所有任务类别均有 substantial 提升空间。我们还对任务设计进行了全面的消融研究，强调了所提任务中精确视频理解的必要性以及当前智能体的局限性。我们希望Ego2Web能成为开发真正强大的AI助手的关键新资源，这些助手能够无缝地在物理世界与数字世界之间观察、理解与行动。
URL
https://arxiv.org/abs/2603.22529
PDF
-
Color When It Counts: Grayscale-Guided Online Triggering for Always-On Streaming Video Sensing
2026-03-23 18:32:36Weitong Cai, Hang Zhang, Yukai Huang, Shitong Sun, Jiankang Deng, Songcen Xu, Jifei Song, Zhensong ZhangAbstract
Always-on sensing is essential for next-generation edge/wearable AI systems, yet continuous high-fidelity RGB video capture remains prohibitively expensive for resource-constrained mobile and edge platforms. We present a new paradigm for efficient streaming video understanding: grayscale-always, color-on-demand. Through preliminary studies, we discover that color is not always necessary. Sparse RGB frames suffice for comparable performance when temporal structure is preserved via continuous grayscale streams. Building on this insight, we propose ColorTrigger, an online training-free trigger that selectively activates color capture based on windowed grayscale affinity analysis. Designed for real-time edge deployment, ColorTrigger uses lightweight quadratic programming to detect chromatic redundancy causally, coupled with credit-budgeted control and dynamic token routing to jointly reduce sensing and inference costs. On streaming video understanding benchmarks, ColorTrigger achieves 91.6% of full-color baseline performance while using only 8.1% RGB frames, demonstrating substantial color redundancy in natural videos and enabling practical always-on video sensing on resource-constrained devices.
Abstract (translated)
持续感知对下一代边缘/可穿戴AI系统至关重要，但连续高保真RGB视频采集对资源受限的移动和边缘平台而言成本过高。我们提出了一种高效流式视频理解的新范式：灰度常开，按需彩色。通过初步研究发现，颜色并非始终必要——当通过连续灰度流保持时间结构时，稀疏的RGB帧即可实现相当的性能。基于这一洞察，我们提出了ColorTrigger，一种无需在线训练的触发机制，该机制通过窗口化灰度相似性分析选择性激活彩色采集。专为实时边缘部署设计，ColorTrigger采用轻量级二次规划进行因果色度冗余检测，并结合信用预算控制和动态令牌路由，共同降低感知与推理成本。在流式视频理解基准测试中，ColorTrigger达到全彩色基线性能的91.6%，而仅使用8.1%的RGB帧，证明了自然视频中存在显著的色彩冗余，并为资源受限设备上的实用持续视频感知提供了可能。
URL
https://arxiv.org/abs/2603.22466
PDF
-
VideoDetective: Clue Hunting via both Extrinsic Query and Intrinsic Relevance for Long Video Understanding
2026-03-23 17:59:51Ruoliu Yang, Chu Wu, Caifeng Shan, Ran He, Chaoyou FuAbstract
Long video understanding remains challenging for multimodal large language models (MLLMs) due to limited context windows, which necessitate identifying sparse query-relevant video segments. However, existing methods predominantly localize clues based solely on the query, overlooking the video's intrinsic structure and varying relevance across segments. To address this, we propose VideoDetective, a framework that integrates query-to-segment relevance and inter-segment affinity for effective clue hunting in long-video question answering. Specifically, we divide a video into various segments and represent them as a visual-temporal affinity graph built from visual similarity and temporal proximity. We then perform a Hypothesis-Verification-Refinement loop to estimate relevance scores of observed segments to the query and propagate them to unseen segments, yielding a global relevance distribution that guides the localization of the most critical segments for final answering with sparse observation. Experiments show our method consistently achieves substantial gains across a wide range of mainstream MLLMs on representative benchmarks, with accuracy improvements of up to 7.5% on VideoMME-long. Our code is available at this https URL
Abstract (translated)
长视频理解对于多模态大语言模型（MLLMs）而言仍是一项挑战，这主要受限于上下文窗口，使得模型必须识别出与查询稀疏相关的视频片段。然而，现有方法大多仅基于查询定位线索，忽视了视频的内在结构以及不同片段间相关性的差异。为解决这一问题，我们提出了VideoDetective框架，该框架整合了查询与片段的相关性以及片段间的关联性，从而在长视频问答中实现高效线索检索。具体而言，我们将视频划分为多个片段，并将其表示为基于视觉相似性与时序邻近性构建的视觉-时序关联图。随后，我们通过假设-验证-优化循环来估算已观测片段与查询的相关性分数，并将这些分数传播至未观测片段，从而获得全局相关性分布，以指导在稀疏观测条件下定位对最终回答最关键的片段。实验表明，我们的方法在多个主流MLLMs和代表性基准测试上均取得了显著提升，在VideoMME-long数据集上的准确率最高提升达7.5%。代码已公开于此链接。
URL
https://arxiv.org/abs/2603.22285
PDF
-
StreamingClaw Technical Report
2026-03-23 15:43:52Jiawei Chen, Zhe Chen, Chaoqun Du, Maokui He, Wei He, Hengtao Li, Qizhen Li, Zide Liu, Hao Ma, Xuhao Pan, Chang Ren, Xudong Rao, Xintian Shen, Chenfeng Wang, Tao Wei, Chengjun Yu, Pengfei Yu, Shengyu Yao, Chunpeng Zhou, Kun Zhan, Lihao Zheng, Pan Zhou, Xuhan Zhu, Yufei ZhengAbstract
Applications such as embodied intelligence rely on a real-time perception-decision-action closed loop, posing stringent challenges for streaming video understanding. However, current agents suffer from fragmented capabilities, such as supporting only offline video understanding, lacking long-term multimodal memory mechanisms, or struggling to achieve real-time reasoning and proactive interaction under streaming inputs. These shortcomings have become a key bottleneck for preventing them from sustaining perception, making real-time decisions, and executing actions in real-world environments. To alleviate these issues, we propose StreamingClaw, a unified agent framework for streaming video understanding and embodied intelligence. It is also an OpenClaw-compatible framework that supports real-time, multimodal streaming interaction. StreamingClaw integrates five core capabilities: (1) It supports real-time streaming reasoning. (2) It supports reasoning about future events and proactive interaction under the online evolution of interaction objectives. (3) It supports multimodal long-term storage, hierarchical evolution, and efficient retrieval of shared memory across multiple agents. (4) It supports a closed-loop of perception-decision-action. In addition to conventional tools and skills, it also provides streaming tools and action-centric skills tailored for real-world physical environments. (5) It is compatible with the OpenClaw framework, allowing it to fully leverage the resources and support of the open-source community. With these designs, StreamingClaw integrates online real-time reasoning, multimodal long-term memory, and proactive interaction within a unified framework. Moreover, by translating decisions into executable actions, it enables direct control of the physical world, supporting practical deployment of embodied interaction.
Abstract (translated)
具身智能等应用依赖于实时感知-决策-行动的闭环，对视频流理解提出了严苛挑战。然而现有智能体存在能力碎片化问题：或仅支持离线视频理解，或缺乏长期多模态记忆机制，或在流式输入下难以实现实时推理与主动交互。这些缺陷已成为阻碍其在真实环境中持续感知、实时决策并执行行动的关键瓶颈。为缓解上述问题，我们提出了StreamingClaw——一个面向流式视频理解与具身智能的统一智能体框架。该框架同时兼容OpenClaw，支持实时多模态流式交互。StreamingClaw整合了五大核心能力：(1) 支持实时流式推理；(2) 支持在交互目标在线演化条件下对未来事件进行推理并实现主动交互；(3) 支持多模态长期存储、分层演化及跨智能体共享记忆的高效检索；(4) 支持感知-决策-行动闭环，除常规工具与技能外，还提供面向真实物理环境的流式工具与以行动为中心的技能；(5) 兼容OpenClaw框架，可充分借助开源社区的资源与支持。基于这些设计，StreamingClaw将在线实时推理、多模态长期记忆与主动交互整合于统一框架，并通过将决策转化为可执行行动，实现对物理世界的直接控制，支持具身交互的实际部署。
URL
https://arxiv.org/abs/2603.22120
PDF
-
Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
2026-03-23 13:15:22Junhao Du, Jialong Xue, Anqi Li, Jincheng Dai, Guo LuAbstract
Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21957
PDF
-
FedCVU: Federated Learning for Cross-View Video Understanding
2026-03-23 07:26:55Shenghan Zhang, Run Ling, Ke Cao, Ao Ma, Zhanjie ZhangAbstract
Federated learning (FL) has emerged as a promising paradigm for privacy-preserving multi-camera video understanding. However, applying FL to cross-view scenarios faces three major challenges: (i) heterogeneous viewpoints and backgrounds lead to highly non-IID client distributions and overfitting to view-specific patterns, (ii) local distribution biases cause misaligned representations that hinder consistent cross-view semantics, and (iii) large video architectures incur prohibitive communication overhead. To address these issues, we propose FedCVU, a federated framework with three components: VS-Norm, which preserves normalization parameters to handle view-specific statistics; CV-Align, a lightweight contrastive regularization module to improve cross-view representation alignment; and SLA, a selective layer aggregation strategy that reduces communication without sacrificing accuracy. Extensive experiments on action understanding and person re-identification tasks under a cross-view protocol demonstrate that FedCVU consistently boosts unseen-view accuracy while maintaining strong seen-view performance, outperforming state-of-the-art FL baselines and showing robustness to domain heterogeneity and communication constraints.
Abstract (translated)
联邦学习（FL）作为一种有前景的隐私保护多摄像头视频理解范式已受到广泛关注。然而，将FL应用于跨视角场景面临三大挑战：（i）异构视角与背景导致客户端数据分布高度非独立同分布，且模型易过拟合至视角特定模式；（ii）本地分布偏差导致表征错位，阻碍跨视角语义一致性；（iii）大型视频架构带来高昂的通信开销。针对上述问题，我们提出FedCVU联邦框架，包含三个核心组件：VS-Norm通过保留归一化参数处理视角特定统计信息；CV-Align作为轻量级对比正则化模块提升跨视角表征对齐；SLA采用选择性层聚合策略在保持精度的同时降低通信成本。在跨视角协议下的动作理解与行人重识别任务中，大量实验表明FedCVU在维持已见视角性能的同时持续提升未见视角准确率，其表现优于现有联邦学习基线方法，并对领域异构性与通信约束展现出强鲁棒性。
URL
https://arxiv.org/abs/2603.21647
PDF
-
CataractSAM-2: A Domain-Adapted Model for Anterior Segment Surgery Segmentation and Scalable Ground-Truth Annotation
2026-03-23 04:40:35Mohammad Eslami, Dhanvinkumar Ganeshkumar, Saber Kazeminasab, Michael G. Morley, Michael V. Boland, Michael M. Lin, John B. Miller, David S. Friedman, Nazlee Zebardast, Lucia Sobrin, Tobias ElzeAbstract
We present CataractSAM-2, a domain-adapted extension of Meta's Segment Anything Model 2, designed for real-time semantic segmentation of cataract ophthalmic surgery videos with high accuracy. Positioned at the intersection of computer vision and medical robotics, CataractSAM-2 enables precise intraoperative perception crucial for robotic-assisted and computer-guided surgical systems. Furthermore, to alleviate the burden of manual labeling, we introduce an interactive annotation framework that combines sparse prompts with video-based mask propagation. This tool significantly reduces annotation time and facilitates the scalable creation of high-quality ground-truth masks, accelerating dataset development for ocular anterior segment surgeries. We also demonstrate the model's strong zero-shot generalization to glaucoma trabeculectomy procedures, confirming its cross-procedural utility and potential for broader surgical applications. The trained model and annotation toolkit are released as open-source resources, establishing CataractSAM-2 as a foundation for expanding anterior ophthalmic surgical datasets and advancing real-time AI-driven solutions in medical robotics, as well as surgical video understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2603.21566
PDF
-
StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding
2026-03-23 02:31:11Guowei Tang, Tianwen Qian, Huanran Zheng, Yifei Wang, Xiaoling WangAbstract
Real-time, continuous understanding of visual signals is essential for real-world interactive AI applications, and poses a fundamental system-level challenge. Existing research on streaming video understanding, however, typically focuses on isolated aspects such as question-answering accuracy under limited visual context or improvements in encoding efficiency, while largely overlooking practical deployability under realistic resource constraints. To bridge this gap, we introduce StreamingEval, a unified evaluation framework for assessing the streaming video understanding capabilities of Video-LLMs under realistic constraints. StreamingEval benchmarks both mainstream offline models and recent online video models under a standardized protocol, explicitly characterizing the trade-off between efficiency, storage and accuracy. Specifically, we adopt a fixed-capacity memory bank to normalize accessible historical visual context, and jointly evaluate visual encoding efficiency, text decoding latency, and task performance to quantify overall system deployability. Extensive experiments across multiple datasets reveal substantial gaps between current Video-LLMs and the requirements of realistic streaming applications, providing a systematic basis for future research in this direction. Codes will be released at this https URL.
Abstract (translated)
实时、连续地理解视觉信号对于现实世界的交互式AI应用至关重要，并构成了一个根本性的系统级挑战。然而，现有关于流式视频理解的研究通常局限于孤立层面，例如在有限视觉上下文下的问答准确性或编码效率的提升，而很大程度上忽视了现实资源约束下的实际可部署性。为弥合这一差距，我们提出了StreamingEval——一个在现实约束下评估视频大语言模型流式视频理解能力的统一评估框架。StreamingEval通过标准化协议对主流离线模型和近期在线视频模型进行基准测试，明确刻画了效率、存储与准确性之间的权衡关系。具体而言，我们采用固定容量的记忆库来规范化可访问的历史视觉上下文，并联合评估视觉编码效率、文本解码延迟及任务性能，以量化整体系统可部署性。在多个数据集上的大量实验表明，当前视频大语言模型与现实流式应用需求之间存在显著差距，为未来该方向研究提供了系统性基础。代码将在此https URL发布。
URL
https://arxiv.org/abs/2603.21493
PDF
-
PEARL: Personalized Streaming Video Understanding Model
2026-03-20 18:47:54Yuanhong Zheng, Ruichuan An, Xiaopeng Lin, Yuxing Liu, Sihan Yang, Huanyu Zhang, Haodong Li, Qintong Zhang, Renrui Zhang, Guopeng Li, Yifan Zhang, Yuheng Li, Wentao ZhangAbstract
Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current multimodal personalization methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model's ability to respond to personalized concepts at exact timestamps under two modes: (1) Frame-level, focusing on a specific person or object in discrete frames, and (2) a novel Video-level, focusing on personalized actions unfolding across continuous frames. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. Concept diversity and annotation quality are strictly ensured through a combined pipeline of automated generation and human verification. To tackle this challenging new setting, we further propose PEARL, a plug-and-play, training-free strategy that serves as a strong baseline. Extensive evaluations across 8 offline and online models demonstrate that PEARL achieves state-of-the-art performance. Notably, it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be a highly effective and robust strategy. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants. Code is available at this https URL.
Abstract (translated)
人类对新概念的认知本质上是一个持续流动的过程：我们不断识别新对象或身份，并随时间更新记忆。然而，当前的多模态个性化方法大多局限于静态图像或离线视频，这割裂了连续视觉输入与即时现实反馈的联系，限制了其为未来AI助手提供实时、交互式个性化响应的能力。为弥合这一差距，我们首次提出并正式定义了新任务——个性化流式视频理解（PSVU）。为促进该新方向的研究，我们引入了PEARL-Bench，这是首个专为该挑战性场景设计的综合基准。它通过两种模式评估模型在精确时间戳对个性化概念的响应能力：（1）帧级模式，关注离散帧中的特定人物或物体；（2）新颖的视频级模式，关注跨连续帧展开的个性化动作。PEARL-Bench包含132个独特视频和2,173条带精确时间戳的细粒度标注。通过自动化生成与人工验证结合的流程，严格确保了概念多样性和标注质量。为应对这一挑战性新场景，我们进一步提出了PEARL——一种即插即用、无需训练的强基线策略。在8个离线与在线模型上的广泛评估表明，PEARL达到了最先进性能。值得注意的是，将其应用于3种不同架构时均能持续提升PSVU效果，证明了其高度有效性与鲁棒性。我们希望本研究能推动视觉-语言模型（VLM）个性化发展，并激发对流式个性化AI助手的进一步研究。代码已公开于此https链接。
URL
https://arxiv.org/abs/2603.20422
PDF
-
VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking
2026-03-20 17:58:47Jingyang Lin, Jialian Wu, Jiang Liu, Ximeng Sun, Ze Wang, Xiaodong Yu, Jiebo Luo, Zicheng Liu, Emad BarsoumAbstract
Video agentic models have advanced challenging video-language tasks. However, most agentic approaches still heavily rely on greedy parsing over densely sampled video frames, resulting in high computational cost. We present VideoSeek, a long-horizon video agent that leverages video logic flow to actively seek answer-critical evidence instead of exhaustively parsing the full video. This insight allows the model to use far fewer frames while maintaining, or even improving, its video understanding capability. VideoSeek operates in a think-act-observe loop with a well-designed toolkit for collecting multi-granular video observations. This design enables query-aware exploration over accumulated observations and supports practical video understanding and reasoning. Experiments on four challenging video understanding and reasoning benchmarks demonstrate that VideoSeek achieves strong accuracy while using far fewer frames than prior video agents and standalone LMMs. Notably, VideoSeek achieves a 10.2 absolute points improvement on LVBench over its base model, GPT-5, while using 93% fewer frames. Further analysis highlights the significance of leveraging video logic flow, strong reasoning capability, and the complementary roles of toolkit design.
Abstract (translated)
视频智能体模型在具有挑战性的视频-语言任务上取得了进展。然而，大多数智能体方法仍严重依赖对密集采样视频帧的贪婪式解析，导致计算成本高昂。本文提出VideoSeek，一个利用视频逻辑流主动寻找答案关键证据而非穷尽解析完整视频的长时序视频智能体。这一洞见使模型能够在保持甚至提升视频理解能力的同时，使用远少于以往的帧数。VideoSeek采用思考-行动-观察循环机制，并配备精心设计的工具包以收集多粒度视频观测。该设计支持对累积观测进行查询感知的探索，并适用于实际的视频理解与推理。在四个具有挑战性的视频理解与推理基准测试上的实验表明，VideoSeek在帧数远少于先前视频智能体及独立大语言模型的情况下，实现了强大的准确率。值得注意的是，VideoSeek在LVBench上相较于其基础模型GPT-5实现了10.2个绝对百分点的提升，而帧数减少93%。进一步分析凸显了利用视频逻辑流、强推理能力以及工具包互补设计的重要性。
URL
https://arxiv.org/abs/2603.20185
PDF
-
Adaptive Greedy Frame Selection for Long Video Understanding
2026-03-20 17:55:32Yuning Huang, Fengqing ZhuAbstract
Large vision--language models (VLMs) are increasingly applied to long-video question answering, yet inference is often bottlenecked by the number of input frames and resulting visual tokens. Naive sparse sampling can miss decisive moments, while purely relevance-driven selection frequently collapses onto near-duplicate frames and sacrifices coverage of temporally distant evidence. We propose a question-adaptive greedy frame selection method that jointly optimizes query relevance and semantic representativeness under a fixed frame budget. Our approach constructs a 1~FPS candidate pool (capped at 1000) with exact timestamp alignment, embeds candidates in two complementary spaces (SigLIP for question relevance and DINOv2 for semantic similarity), and selects frames by greedily maximizing a weighted sum of a modular relevance term and a facility-location coverage term. This objective is normalized, monotone, and submodular, yielding a standard (1-1/e) greedy approximation guarantee. To account for question-dependent trade-offs between relevance and coverage, we introduce four preset strategies and a lightweight text-only question-type classifier that routes each query to its best-performing preset. Experiments on MLVU show consistent accuracy gains over uniform sampling and a strong recent baseline across frame budgets, with the largest improvements under tight budgets.
Abstract (translated)
大型视觉-语言模型（VLMs）正日益应用于长视频问答任务，但其推理过程常因输入帧数及由此产生的视觉标记数量而受限。简单的稀疏采样可能遗漏关键片段，而纯粹基于相关性的选择则容易陷入近重复帧，并牺牲时间上相距较远的证据覆盖。我们提出一种问题自适应的贪心帧选择方法，在固定帧数预算下联合优化查询相关性与语义代表性。该方法构建一个精确对齐时间戳的1FPS候选池（上限1000帧），将候选帧嵌入两个互补空间（SigLIP用于问题相关性，DINOv2用于语义相似性），并通过贪心最大化一个加权和来选择帧，该加权和由模块化相关性项与设施位置覆盖项组成。此目标函数具有归一化、单调和子模特性，因此享有标准(1-1/e)贪心近似保证。为适应不同问题对相关性与覆盖的权衡需求，我们引入四种预设策略及一个轻量级纯文本问题类型分类器，将每个查询路由至其表现最佳的预设策略。在MLVU上的实验表明，相比均匀采样和近期强基线，我们的方法在各类帧数预算下均持续提升准确率，且在预算紧张时改进最为显著。
URL
https://arxiv.org/abs/2603.20180
PDF
-
Leum-VL Technical Report
2026-03-20 12:48:26Yuxuan He, Chaiming Huang, Yifan Wu, Hongjun Wang, Chenkui Shen, Jifan Zhang, Long LiAbstract
A short video succeeds not simply because of what it shows, but because of how it schedules attention -- yet current multimodal models lack the structural grammar to parse or produce this organization. Existing models can describe scenes, answer event-centric questions, and read on-screen text, but they are far less reliable at identifying timeline-grounded units such as hooks, cut rationales, shot-induced tension, and platform-facing packaging cues. We propose SV6D (Structured Video in Six Dimensions), inspired by professional storyboard practice in film and television production, a representation framework that decomposes internet-native video into six complementary structural dimensions -- subject, aesthetics, camera language, editing, narrative, and dissemination -- with each label tied to physically observable evidence on the timeline. We formalize a unified optimization objective over SV6D that combines Hungarian-matched temporal alignment, dimension-wise semantic label distance, and quality regularization. Building on this framework, we present Leum-VL-8B, an 8B video-language model that realizes the SV6D objective through an expert-driven post-training pipeline, further refined through verifiable reinforcement learning on perception-oriented tasks. Leum-VL-8B achieves 70.8 on VideoMME (w/o subtitles), 70.0 on MVBench, and 61.6 on MotionBench, while remaining competitive on general multimodal evaluations such as MMBench-EN. We also construct FeedBench, a benchmark for structure-sensitive short-video understanding. Our results indicate that the missing layer in video AI is not pixel generation but structural representation: grounded on the timeline, linked to visible evidence, and directly consumable by downstream workflows such as editing, retrieval, recommendation, and generation control, including text-heavy internet video formats with overlays and image-text layouts.
Abstract (translated)
短视频的成功不仅在于其展示的内容，更在于其如何调度注意力——然而当前多模态模型缺乏解析或生成这种组织所需的结构性语法。现有模型能够描述场景、回答以事件为中心的问题并阅读屏幕文本，但在识别基于时间线的单元（如钩子、剪辑依据、镜头引发的张力以及面向平台的包装信号）方面远不可靠。我们提出SV6D（六维结构化视频），其灵感来自影视制作中的专业故事板实践，该表示框架将互联网原生视频分解为六个互补的结构维度——主体、美学、镜头语言、剪辑、叙事与传播——每个标签均与时间线上可观察的物理证据相关联。我们形式化了结合匈牙利匹配的时间对齐、维度级语义标签距离和质量正则化的统一优化目标。基于此框架，我们推出了Leum-VL-8B，一个8B参数的视频-语言模型，它通过专家驱动的后训练流水线实现了SV6D目标，并进一步通过面向感知任务的可验证强化学习进行精炼。Leum-VL-8B在VideoMME（无字幕）上达到70.8，在MVBench上达到70.0，在MotionBench上达到61.6，同时在MMBench-EN等通用多模态评估中仍具竞争力。我们还构建了FeedBench，一个对结构敏感的短视频理解基准。我们的结果表明，视频AI中缺失的层面并非像素生成，而是结构性表示：它基于时间线、关联可见证据，并可直接被编辑、检索、推荐和生成控制等下游工作流消费，包括带有叠加层和图文排版的文本密集型互联网视频格式。
URL
https://arxiv.org/abs/2603.20354
PDF
-
ParallelVLM: Lossless Video-LLM Acceleration with Visual Alignment Aware Parallel Speculative Decoding
2026-03-20 03:30:32Quan Kong, Yuhao Shen, Yicheng Ji, Huan Li, Cong WangAbstract
Although current Video-LLMs achieve impressive performance in video understanding tasks, their autoregressive decoding efficiency remains constrained by the massive number of video tokens. Visual token pruning can partially ease this bottleneck, yet existing approaches still suffer from information loss and yield only modest acceleration in decoding. In this paper, we propose ParallelVLM, a training-free draft-then-verify speculative decoding framework that overcomes both mutual waiting and limited speedup-ratio problems between draft and target models in long-video settings. ParallelVLM features two parallelized stages that maximize hardware utilization and incorporate an Unbiased Verifier-Guided Pruning strategy to better align the draft and target models by eliminating the positional bias in attention-guided pruning. Extensive experiments demonstrate that ParallelVLM effectively expands the draft window by $1.6\sim1.8\times$ with high accepted lengths, and accelerates various video understanding benchmarks by 3.36$\times$ on LLaVA-Onevision-72B and 2.42$\times$ on Qwen2.5-VL-32B compared with vanilla autoregressive decoding.
Abstract (translated)
尽管当前视频大语言模型在视频理解任务中表现出色，但其自回归解码效率仍受海量视频标记的制约。视觉标记剪枝虽能部分缓解这一瓶颈，现有方法仍存在信息损失问题，且解码加速效果有限。本文提出ParallelVLM，一种无需训练的草稿-验证推测性解码框架，可解决长视频场景下草稿模型与目标模型间的相互等待及加速比受限问题。该框架采用两阶段并行设计以最大化硬件利用率，并引入无偏验证器引导剪枝策略，通过消除注意力引导剪枝中的位置偏差来更好对齐草稿与目标模型。大量实验表明，ParallelVLM在保持高接受长度的同时，将草稿窗口有效扩展1.6~1.8倍，并在LLaVA-Onevision-72B和Qwen2.5-VL-32B上分别实现3.36倍与2.42倍加速，显著优于原生自回归解码。
URL
https://arxiv.org/abs/2603.19610
PDF
-
CurveStream: Boosting Streaming Video Understanding in MLLMs via Curvature-Aware Hierarchical Visual Memory Management
2026-03-20 02:28:57Chao Wang, Xudong Tan, Jianjian Cao, Kangcong Li, Tao ChenAbstract
Multimodal Large Language Models have achieved significant success in offline video understanding, yet their application to streaming videos is severely limited by the linear explosion of visual tokens, which often leads to Out-of-Memory (OOM) errors or catastrophic forgetting. Existing visual retention and memory management methods typically rely on uniform sampling, low-level physical metrics, or passive cache eviction. However, these strategies often lack intrinsic semantic awareness, potentially disrupting contextual coherence and blurring transient yet critical semantic transitions. To address these limitations, we propose CurveStream, a training-free, curvature-aware hierarchical visual memory management framework. Our approach is motivated by the key observation that high-curvature regions along continuous feature trajectories closely align with critical global semantic transitions. Based on this geometric insight, CurveStream evaluates real-time semantic intensity via a Curvature Score and integrates an online K-Sigma dynamic threshold to adaptively route frames into clear and fuzzy memory states under a strict token budget. Evaluations across diverse temporal scales confirm that this lightweight framework, CurveStream, consistently yields absolute performance gains of over 10% (e.g., 10.69% on StreamingBench and 13.58% on OVOBench) over respective baselines, establishing new state-of-the-art results for streaming video this http URL code will be released at this https URL.
Abstract (translated)
多模态大语言模型在离线视频理解中已取得显著成功，但其在流式视频上的应用却因视觉token的线性爆炸而受到严重限制，这常导致内存溢出(OOM)错误或灾难性遗忘。现有的视觉保留与内存管理方法通常依赖均匀采样、低层物理指标或被动的缓存驱逐策略。然而，这些策略往往缺乏内在的语义感知能力，可能破坏上下文连贯性，并模糊短暂却关键的语义转折。为应对这些局限，我们提出CurveStream——一个无需训练、具备曲率感知的层次化视觉内存管理框架。该方法源于一个关键观察：连续特征轨迹上的高曲率区域与关键的全局语义转折高度吻合。基于这一几何洞察，CurveStream通过曲率评分评估实时语义强度，并整合在线K-Sigma动态阈值，在严格token预算下将帧自适应路由至清晰与模糊内存状态。跨多样时间尺度的评估证实，这一轻量级框架CurveStream相较于各基线模型持续取得超过10%的绝对性能提升（例如在StreamingBench上提升10.69%，在OVOBench上提升13.58%），为流式视频理解建立了新的最先进结果。代码将在此链接发布。
URL
https://arxiv.org/abs/2603.19571
PDF
-
LVOmniBench: Pioneering Long Audio-Video Understanding Evaluation for Omnimodal LLMs
2026-03-19 17:58:13Keda Tao, Yuhua Zheng, Jia Xu, Wenjie Du, Kele Shao, Hesong Wang, Xueyi Chen, Xin Jin, Junhan Zhu, Bohan Yu, Weiqiang Wang, Jian Liu, Can Qin, Yulun Zhang, Ming-Hsuan Yang, Huan WangAbstract
Recent advancements in omnimodal large language models (OmniLLMs) have significantly improved the comprehension of audio and video inputs. However, current evaluations primarily focus on short audio and video clips ranging from 10 seconds to 5 minutes, failing to reflect the demands of real-world applications, where videos typically run for tens of minutes. To address this critical gap, we introduce LVOmniBench, a new benchmark designed specifically for the cross-modal comprehension of long-form audio and video. This dataset comprises high-quality videos sourced from open platforms that feature rich audio-visual dynamics. Through rigorous manual selection and annotation, LVOmniBench comprises 275 videos, ranging in duration from 10 to 90 minutes, and 1,014 question-answer (QA) pairs. LVOmniBench aims to rigorously evaluate the capabilities of OmniLLMs across domains, including long-term memory, temporal localization, fine-grained understanding, and multimodal perception. Our extensive evaluation reveals that current OmniLLMs encounter significant challenges when processing extended audio-visual inputs. Open-source models generally achieve accuracies below 35%, whereas the Gemini 3 Pro reaches a peak accuracy of approximately 65%. We anticipate that this dataset, along with our empirical findings, will stimulate further research and the development of advanced models capable of resolving complex cross-modal understanding problems within long-form audio-visual contexts.
Abstract (translated)
URL
https://arxiv.org/abs/2603.19217
PDF
-
TAU-R1: Visual Language Model for Traffic Anomaly Understanding
2026-03-19 16:23:21Yuqiang Lin, Kehua Chen, Sam Lockyer, Arjun Yadav, Mingxuan Sui, Shucheng Zhang, Yan Shi, Bingzhang Wang, Yuang Zhang, Markus Zarbock, Florain Stanek, Adrian Evans, Wenbin Li, Yinhai Wang, Nic ZhangAbstract
Traffic Anomaly Understanding (TAU) is important for traffic safety in Intelligent Transportation Systems. Recent vision-language models (VLMs) have shown strong capabilities in video understanding. However, progress on TAU remains limited due to the lack of benchmarks and task-specific methodologies. To address this limitation, we introduce Roundabout-TAU, a dataset constructed from real-world roundabout videos collected in collaboration with the City of Carmel, Indiana. The dataset contains 342 clips and is annotated with more than 2,000 question-answer pairs covering multiple aspects of traffic anomaly understanding. Building on this benchmark, we propose TAU-R1, a two-layer vision-language framework for TAU. The first layer is a lightweight anomaly classifier that performs coarse anomaly categorisation, while the second layer is a larger anomaly reasoner that generates detailed event summaries. To improve task-specific reasoning, we introduce a two-stage training strategy consisting of decomposed-QA-enhanced supervised fine-tuning followed by TAU-GRPO, a GRPO-based post-training method with TAU-specific reward functions. Experimental results show that TAU-R1 achieves strong performance on both anomaly classification and reasoning tasks while maintaining deployment efficiency. The dataset and code are available at: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.19098
PDF
-
Em-Garde: A Propose-Match Framework for Proactive Streaming Video Understanding
2026-03-19 15:47:50Yikai Zheng, Xin Ding, Yifan Yang, Shiqi Jiang, Hao Wu, Qianxi Zhang, Weijun Wang, Ting Cao, Yunxin LiuAbstract
Recent advances in Streaming Video Understanding has enabled a new interaction paradigm where models respond proactively to user queries. Current proactive VideoLLMs rely on per-frame triggering decision making, which suffers from an efficiency-accuracy dilemma. We propose Em-Garde, a novel framework that decouples semantic understanding from streaming perception. At query time, the Instruction-Guided Proposal Parser transforms user queries into structured, perceptually grounded visual proposals; during streaming, a Lightweight Proposal Matching Module performs efficient embedding-based matching to trigger responses. Experiments on StreamingBench and OVO-Bench demonstrate consistent improvements over prior models in proactive response accuracy and efficiency, validating an effective solution for proactive video understanding under strict computational constraints.
Abstract (translated)
URL
https://arxiv.org/abs/2603.19054
PDF
-
Motion-o: Trajectory-Grounded Video Reasoning
2026-03-19 13:00:29Bishoy Galoaa, Shayda Moezzi, Xiangyu Bai, Sarah OstadabbasAbstract
Recent research has made substantial progress on video reasoning, with many models leveraging spatio-temporal evidence chains to strengthen their inference capabilities. At the same time, a growing set of datasets and benchmarks now provides structured annotations designed to support and evaluate such reasoning. However, little attention has been paid to reasoning about \emph{how} objects move between observations: no prior work has articulated the motion patterns by connecting successive observations, leaving trajectory understanding implicit and difficult to verify. We formalize this missing capability as Spatial-Temporal-Trajectory (STT) reasoning and introduce \textbf{Motion-o}, a motion-centric video understanding extension to visual language models that makes trajectories explicit and verifiable. To enable motion reasoning, we also introduce a trajectory-grounding dataset artifact that expands sparse keyframe supervision via augmentation to yield denser bounding box tracks and a stronger trajectory-level training signal. Finally, we introduce Motion Chain of Thought (MCoT), a structured reasoning pathway that makes object trajectories through discrete \texttt{<motion/>} tag summarizing per-object direction, speed, and scale (of velocity) change to explicitly connect grounded observations into trajectories. To train Motion-o, we design a reward function that compels the model to reason directly over visual evidence, all while requiring no architectural modifications. Empirical results demonstrate that Motion-o improves spatial-temporal grounding and trajectory prediction while remaining fully compatible with existing frameworks, establishing motion reasoning as a critical extension for evidence-based video understanding. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.18856
PDF
-
VideoAtlas: Navigating Long-Form Video in Logarithmic Compute
2026-03-18 17:20:19Mohamed Eltahir, Ali Habibullah, Yazan Alshoibi, Lama Ayash, Tanveer Hussain, Naeemullah KhanAbstract
Extending language models to video introduces two challenges: representation, where existing methods rely on lossy approximations, and long-context, where caption- or agent-based pipelines collapse video into text and lose visual fidelity. To overcome this, we introduce \textbf{VideoAtlas}, a task-agnostic environment to represent video as a hierarchical grid that is simultaneously lossless, navigable, scalable, caption- and preprocessing-free. An overview of the video is available at a glance, and any region can be recursively zoomed into, with the same visual representation used uniformly for the video, intermediate investigations, and the agent's memory, eliminating lossy text conversion end-to-end. This hierarchical structure ensures access depth grows only logarithmically with video length. For long-context, Recursive Language Models (RLMs) recently offered a powerful solution for long text, but extending them to visual domain requires a structured environment to recurse into, which \textbf{VideoAtlas} provides. \textbf{VideoAtlas} as a Markov Decision Process unlocks Video-RLM: a parallel Master-Worker architecture where a Master coordinates global exploration while Workers concurrently drill into assigned regions to accumulate lossless visual evidence. We demonstrate three key findings: (1)~logarithmic compute growth with video duration, further amplified by a 30-60\% multimodal cache hit rate arising from the grid's structural reuse. (2)~environment budgeting, where bounding the maximum exploration depth provides a principled compute-accuracy hyperparameter. (3)~emergent adaptive compute allocation that scales with question granularity. When scaling from 1-hour to 10-hour benchmarks, Video-RLM remains the most duration-robust method with minimal accuracy degradation, demonstrating that structured environment navigation is a viable and scalable paradigm for video understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2603.17948
PDF
