Video_Caption
-
AURA: Always-On Understanding and Real-Time Assistance via Video Streams
2026-04-05 16:53:46Xudong Lu, Yang Bo, Jinpeng Chen, Shuhan Li, Xintong Guo, Huankang Guan, Fang Liu, Dunyuan Xu, Peiwen Sun, Heyang Sun, Rui Liu, Hongsheng LiAbstract
Video Large Language Models (VideoLLMs) have achieved strong performance on many video understanding tasks, but most existing systems remain offline and are not well-suited for live video streams that require continuous observation and timely response. Recent streaming VideoLLMs have made progress, yet current approaches often rely on decoupled trigger-response pipelines or are limited to captioning-style narration, reducing their effectiveness for open-ended question answering and long-horizon interaction. We propose AURA (Always-On Understanding and Real-Time Assistance), an end-to-end streaming visual interaction framework that enables a unified VideoLLM to continuously process video streams and support both real-time question answering and proactive responses. AURA integrates context management, data construction, training objectives, and deployment optimization for stable long-horizon streaming interaction. It achieves state-of-the-art performance on streaming benchmarks and supports a real-time demo system with ASR and TTS running at 2 FPS on two 80G accelerators. We release the AURA model together with a real-time inference framework to facilitate future research.
Abstract (translated)
URL
https://arxiv.org/abs/2604.04184
PDF
-
SGTA: Scene-Graph Based Multi-Modal Traffic Agent for Video Understanding
2026-04-04 12:01:49Xingcheng Zhou, Mingyu Liu, Walter Zimmer, Jiajie Zhang, Alois KnollAbstract
We present Scene-Graph Based Multi-Modal Traffic Agent (SGTA), a modular framework for traffic video understanding that combines structured scene graphs with multi-modal reasoning. It constructs a traffic scene graph from roadside videos using detection, tracking, and lane extraction, followed by tool-based reasoning over both symbolic graph queries and visual inputs. SGTA adopts ReAct to process interleaved reasoning traces from large language models with tool invocations, enabling interpretable decision-making for complex video questions. Experiments on selected TUMTraffic VideoQA dataset sample demonstrate that SGTA achieves competitive accuracy across multiple question types while providing transparent reasoning steps. These results highlight the potential of integrating structured scene representations with multi-modal agents for traffic video understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2604.03697
PDF
-
KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Models
2026-04-03 19:18:22Haifeng Huang, Yang LiAbstract
Video Large Language Models (Video LLMs) achieve strong performance on video understanding tasks but suffer from high inference costs due to the large number of visual tokens. We propose KiToke, a training-free, query-agnostic token compression approach that reduces spatiotemporal redundancy while preserving critical visual information. Our method estimates token diversity globally using a kernel-based redundancy measure, enabling content-adaptive selection that remains effective under extreme token budgets, and further introduces a lightweight temporal interval construction with interval-aware token merging to maintain temporal coherence. Unlike prior methods that rely on local or segment-level heuristics, KiToke explicitly captures global redundancy across an entire video, leading to more efficient token utilization. Extensive experiments on multiple video understanding benchmarks and Video LLM backbones demonstrate that KiToke consistently outperforms existing training-free compression methods, with particularly large gains at aggressive retention ratios down to 1%.
Abstract (translated)
视频大语言模型（Video LLMs）在视频理解任务上表现出色，但由于视觉标记数量庞大，导致推理成本高昂。本文提出KiToke，一种无需训练、与查询无关的标记压缩方法，可在保留关键视觉信息的同时减少时空冗余。该方法采用基于核的冗余度测量，从全局估计标记多样性，实现内容自适应选择，即使在极端标记预算下仍保持有效性，并进一步引入轻量级的时间间隔构建与间隔感知标记合并机制，以维持时间连贯性。与依赖局部或片段级启发式规则的先前方法不同，KiToke能够显式捕捉整个视频的全局冗余，从而实现更高效的标记利用。在多个视频理解基准测试及Video LLM主干网络上的大量实验表明，KiToke持续优于现有无需训练的压缩方法，尤其在低至1%的保留率下实现显著性能提升。
URL
https://arxiv.org/abs/2604.03414
PDF
-
Progressive Video Condensation with MLLM Agent for Long-form Video Understanding
2026-04-03 09:00:38Yufei Yin, Yuchen Xing, Qianke Meng, Minghao Chen, Yan Yang, Zhou YuAbstract
Understanding long videos requires extracting query-relevant information from long sequences under tight compute budgets. Existing text-then-LLM pipelines lose fine-grained visual cues, while video-based multimodal large language models (MLLMs) can keep visual details but are too frame-hungry and computationally expensive. In this work, we aim to harness MLLMs for efficient video understanding. We propose ProVCA, a progressive video condensation agent that iteratively locates key video frames at multiple granularities. ProVCA first adopts a segment localization module to identify the video segment relevant to the query, then a snippet selection module to select important snippets based on similarity, and finally a keyframe refinement module to pinpoint specific keyframes in those snippets. By progressively narrowing the scope from coarse segments to fine frames, ProVCA identifies a small set of keyframes for MLLM-based reasoning. ProVCA achieves state-of-the-art zero-shot accuracies of 69.3\% on EgoSchema, 80.5\% on NExT-QA, and 77.7\% on IntentQA, while using fewer frames than previous training-free methods.
Abstract (translated)
理解长视频需要在有限的计算预算下，从长序列中提取与查询相关的信息。现有的“文本处理+大语言模型”流程会丢失细粒度的视觉线索，而基于视频的多模态大语言模型（MLLMs）虽能保留视觉细节，却因帧数需求过高而计算成本昂贵。本研究旨在利用MLLMs实现高效视频理解。我们提出了ProVCA——一种渐进式视频压缩智能体，可迭代地在多种粒度下定位关键视频帧。ProVCA首先采用片段定位模块识别与查询相关的视频片段，接着通过片段选择模块基于相似度筛选重要片段，最终通过关键帧精炼模块在片段内确定具体关键帧。通过从粗粒度片段到细粒度帧逐步缩小范围，ProVCA为基于MLLM的推理识别出少量关键帧。在无需训练的条件下，ProVCA在EgoSchema上达到69.3%、NExT-QA上80.5%、IntentQA上77.7%的零样本准确率，且使用的帧数少于此前无训练方法。
URL
https://arxiv.org/abs/2604.02891
PDF
-
A Simple Baseline for Streaming Video Understanding
2026-04-02 17:58:04Yujiao Shen, Shulin Tian, Jingkang Yang, Ziwei LiuAbstract
Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.
Abstract (translated)
URL
https://arxiv.org/abs/2604.02317
PDF
-
GroundVTS: Visual Token Sampling in Multimodal Large Language Models for Video Temporal Grounding
2026-04-02 14:19:58Rong Fan, Kaiyan Xiao, Minghao Zhu, Liuyi Wang, Kai Dai, Zhao YangAbstract
Video temporal grounding (VTG) is a critical task in video understanding and a key capability for extending video large language models (Vid-LLMs) to broader applications. However, existing Vid-LLMs rely on uniform frame sampling to extract video information, resulting in a sparse distribution of key frames and the loss of crucial temporal cues. To address this limitation, we propose Grounded Visual Token Sampling (GroundVTS), a Vid-LLM architecture that focuses on the most informative temporal segments. GroundVTS employs a fine-grained, query-guided mechanism to filter visual tokens before feeding them into the LLM, thereby preserving essential spatio-temporal information and maintaining temporal coherence. Futhermore, we introduce a progressive optimization strategy that enables the LLM to effectively adapt to the non-uniform distribution of visual features, enhancing its ability to model temporal dependencies and achieve precise video localization. We comprehensively evaluate GroundVTS on three standard VTG benchmarks, where it outperforms existing methods, achieving a 7.7-point improvement in mIoU for moment retrieval and 12.0-point improvement in mAP for highlight detection. Code is available at this https URL.
Abstract (translated)
视频时间定位（VTG）是视频理解中的关键任务，也是扩展视频大语言模型（Vid-LLMs）应用范围的核心能力。然而，现有Vid-LLMs依赖均匀帧采样提取视频信息，导致关键帧分布稀疏并丢失重要时间线索。针对这一局限，我们提出了Grounded Visual Token Sampling（GroundVTS），一种聚焦于最具信息量时间片段的Vid-LLM架构。GroundVTS采用细粒度、查询引导的机制，在将视觉标记输入大语言模型前进行筛选，从而保留关键时空信息并维持时间连贯性。此外，我们引入了渐进式优化策略，使大语言模型能够有效适应视觉特征的非均匀分布，增强其建模时间依赖关系的能力，实现精准的视频定位。我们在三个标准VTG基准测试上对GroundVTS进行了全面评估，其性能超越现有方法，在时刻检索任务中mIoU提升7.7点，在亮点检测任务中mAP提升12.0点。代码已公开于此https链接。
URL
https://arxiv.org/abs/2604.02093
PDF
-
HieraVid: Hierarchical Token Pruning for Fast Video Large Language Models
2026-04-02 10:40:15Yansong Guo, Chaoyang Zhu, Jiayi Ji, Jianghang Lin, Liujuan CaoAbstract
Video Large Language Models (VideoLLMs) have demonstrated impressive capabilities in video understanding, yet the massive number of input video tokens incurs a significant computational burden for deployment. Existing methods mainly prune video tokens at input level while neglecting the inherent information structure embedded in videos and large language models (LLMs). To address this, we propose HieraVid, a hierarchical pruning framework that progressively and dynamically reduces visual redundancy. Based on two observations that videos possess the segment-frame structure and LLMs internally propagate multi-modal information unidirectionally, we decompose pruning into three levels: 1) segment-level, where video tokens are first temporally segmented and spatially merged; 2) frame-level, where similar frames within the same segment are jointly pruned to preserve diversity; 3) layer-level, redundancy gradually shrinks as LLM layer increases w/o compromising performance. We conduct extensive experiments on four widely used video understanding benchmarks to comprehensively evaluate the effectiveness of HieraVid. Remarkably, with only 30% of tokens retained, HieraVid achieves new state-of-the-art performance, while maintaining over 98% and 99% of the performance of LLaVA-Video-7B and LLaVA-OneVision-7B, respectively.
Abstract (translated)
视频大语言模型（VideoLLMs）在视频理解方面展现出卓越能力，但海量的输入视频令牌给实际部署带来了显著的计算负担。现有方法主要在输入层面对视频令牌进行剪枝，却忽视了视频与大语言模型（LLMs）本身蕴含的信息结构。为此，我们提出HieraVid——一个分层剪枝框架，通过渐进式动态方式降低视觉冗余。基于视频具有段-帧结构、LLMs内部以单向方式传播多模态信息这两个观察，我们将剪枝分解为三个层次：1）段级，首先对视频令牌进行时间分段与空间合并；2）帧级，在同一段内联合剪除相似帧以保留多样性；3）层级，随着LLM层数增加，冗余度逐渐降低且不影响性能。我们在四个广泛使用的视频理解基准上开展大量实验，全面评估HieraVid的有效性。值得注意的是，HieraVid仅保留30%的令牌即达到新的最优性能，同时分别保持LLaVA-Video-7B与LLaVA-OneVision-7B超过98%与99%的性能表现。
URL
https://arxiv.org/abs/2604.01881
PDF
-
VideoZeroBench: Probing the Limits of Video MLLMs with Spatio-Temporal Evidence Verification
2026-04-02 03:29:43Jiahao Meng, Tan Yue, Qi Xu, Haochen Wang, Zhongwei Ren, Weisong Liu, Yuhao Wang, Renrui Zhang, Yunhai Tong, Haodong DuanAbstract
Recent video multimodal large language models achieve impressive results across various benchmarks. However, current evaluations suffer from two critical limitations: (1) inflated scores can mask deficiencies in fine-grained visual understanding and reasoning, and (2) answer correctness is often measured without verifying whether models identify the precise spatio-temporal evidence supporting their predictions. To address this, we present VideoZeroBench, a hierarchical benchmark designed for challenging long-video question answering that rigorously verifies spatio-temporal evidence. It comprises 500 manually annotated questions across 13 domains, paired with temporal intervals and spatial bounding boxes as evidence. To disentangle answering generation, temporal grounding, and spatial grounding, we introduce a five-level evaluation protocol that progressively tightens evidence requirements. Experiments show that even Gemini-3-Pro correctly answers fewer than 17% of questions under the standard end-to-end QA setting (Level-3). When grounding constraints are imposed, performance drops sharply: No model exceeds 1% accuracy when both correct answering and accurate spatio-temporal localization are required (Level-5), with most failing to achieve any correct grounded predictions. These results expose a significant gap between surface-level answer correctness and genuine evidence-based reasoning, revealing that grounded video understanding remains a bottleneck for long-video QA. We further analyze performance across minimal evidence spans, atomic abilities, and inference paradigms, providing insights for future research in grounded video reasoning. The benchmark and code will be made publicly available.
Abstract (translated)
近期视频多模态大语言模型在各种基准测试中取得了令人瞩目的成果。然而，当前评估存在两个关键局限：(1) 虚高的分数可能掩盖模型在细粒度视觉理解与推理方面的缺陷；(2) 答案正确性的判定往往未验证模型是否识别出支撑其预测的精确时空证据。为解决此问题，我们提出了 VideoZeroBench——一个面向挑战性长视频问答的层级化基准，可严格验证时空证据。该基准包含13个领域中500个手工标注的问题，并配以时间区间和空间边界框作为证据。为解耦答案生成、时序定位与空间定位，我们设计了五级评估协议，逐步收紧证据要求。实验表明，即使在标准端到端问答设置（Level-3）下，Gemini-3-Pro 正确回答的问题比例仍不足17%。当施加定位约束时，性能急剧下降：在同时要求答案正确与时空定位准确（Level-5）的条件下，所有模型准确率均未超过1%，多数模型甚至无法产生任何正确的定位预测。这些结果揭示了表面答案正确性与真实基于证据的推理之间存在显著差距，表明基于定位的视频理解仍是长视频问答的瓶颈。我们还针对最小证据跨度、原子能力及推理范式进行了性能分析，为未来基于定位的视频推理研究提供见解。该基准及代码将公开发布。
URL
https://arxiv.org/abs/2604.01569
PDF
-
Reinforcing Consistency in Video MLLMs with Structured Rewards
2026-04-01 23:15:34Yihao Quan, Zeru Shi, Jinman Zhao, Ruixiang TangAbstract
Multimodal large language models (MLLMs) have achieved remarkable progress in video understanding. However, seemingly plausible outputs often suffer from poor visual and temporal grounding: a model may fabricate object existence, assign incorrect attributes, or collapse repeated events while still producing a globally reasonable caption or answer. We study this failure mode through a compositional consistency audit that decomposes a caption into supporting factual and temporal claims, investigating whether a correct high-level prediction is actually backed by valid lower-level evidence. Our top-down audit reveals that even correct root relational claims often lack reliable attribute and existence support. This indicates that standard sentence-level supervision is a weak proxy for faithful video understanding. Furthermore, when turning to reinforcement learning (RL) for better alignment, standard sentence-level rewards often prove too coarse to accurately localize specific grounding failures. To address this, we replace generic sentence-level rewards with a structured reward built from factual and temporal units. Our training objective integrates three complementary components: (1) an instance-aware scene-graph reward for factual objects, attributes, and relations; (2) a temporal reward for event ordering and repetition; and (3) a video-grounded VQA reward for hierarchical self-verification. Across temporal, general video understanding, and hallucination-oriented benchmarks, this objective yields consistent gains on open-source backbones. These results suggest that structured reward shaping is a practical route to more faithful video understanding.
Abstract (translated)
多模态大语言模型（MLLMs）在视频理解领域已取得显著进展。然而，看似合理的输出往往存在视觉与时间定位不准确的问题：模型可能虚构物体存在、赋予错误属性，或在重复事件处理上出现偏差，却仍生成整体通顺的说明或回答。本研究通过组合一致性审计方法，将说明分解为支撑性事实主张与时间主张，探究高层预测是否真正具备有效的底层证据支持。这种自上而下的审计揭示，即使正确的关系性主张也常缺乏可靠的属性与存在性支撑，表明标准句子级监督对忠实视频理解而言是较弱的代理指标。此外，当转向强化学习（RL）以优化对齐时，标准句子级奖励往往过于粗糙，难以精确定位具体定位失败。为此，我们采用基于事实与时间单元的结构化奖励替代通用句子级奖励。训练目标整合三个互补组件：（1）面向实例的场景图奖励，用于事实性对象、属性与关系；（2）针对事件排序与重复的时间奖励；（3）用于分层自验证的视频基础VQA奖励。在时间理解、通用视频理解及面向幻觉的基准测试中，该目标在开源模型上均实现稳定提升。结果表明，结构化奖励塑造是提升视频理解忠实度的可行路径。
URL
https://arxiv.org/abs/2604.01460
PDF
-
Query-Conditioned Evidential Keyframe Sampling for MLLM-Based Long-Form Video Understanding
2026-04-01 15:02:50Yiheng Wang, Lichen Zhu, Yueqian Lin, Yudong Liu, Jingyang Zhang, Hai "Helen" Li, Yiran ChenAbstract
Multimodal Large Language Models (MLLMs) have shown strong performance on video question answering, but their application to long-form videos is constrained by limited context length and computational cost, making keyframe sampling essential. Existing approaches typically rely on semantic relevance or reinforcement learning, which either fail to capture evidential clues or suffer from inefficient combinatorial optimization. In this work, we propose an evidence-driven keyframe sampling framework grounded in information bottleneck theory. We formulate keyframe selection as maximizing the conditional mutual information between selected frames and the query, providing a principled objective that reflects each frame's contribution to answering the question. To make this objective tractable, we exploit its structure to derive a decomposed optimization that reduces subset selection to independent frame-level scoring. We further introduce a query-conditioned evidence scoring network trained with a contrastive objective to estimate evidential importance efficiently. Experiments on long-form video understanding benchmarks show that our method consistently outperforms prior sampling strategies under strict token budgets, while significantly improving training efficiency.
Abstract (translated)
多模态大语言模型（MLLMs）在视频问答任务中展现出卓越性能，但其应用于长视频时受限于上下文长度与计算成本，使得关键帧采样成为必要环节。现有方法通常依赖语义相关性或强化学习，前者难以捕捉证据性线索，后者则面临组合优化效率低下的问题。本研究提出一个基于信息瓶颈理论的证据驱动关键帧采样框架，将关键帧选择建模为最大化所选帧与查询之间的条件互信息，从而建立能反映各帧对回答问题贡献的原则性目标。为实现该目标的可计算性，我们利用其结构特性推导出分解优化方案，将子集选择问题转化为独立的帧级评分。进一步引入经对比学习目标训练的查询条件证据评分网络，以高效估计帧的证据重要性。在长视频理解基准测试中，本方法在严格令牌预算下持续优于现有采样策略，同时显著提升训练效率。
URL
https://arxiv.org/abs/2604.01002
PDF
-
Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment
2026-04-01 13:55:28Zhuchenyang Liu, Yao Zhang, Yu XiaoAbstract
2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect errors, and provide step-by-step guidance. In mixed reality settings, such systems must recognize completed and ongoing steps from the camera feed and align them with the diagram instructions. Vision Language Models (VLMs) show promise for this task, but face a depiction gap because assembly diagrams and video frames share few visual features. To systematically assess this gap, we construct IKEA-Bench, a benchmark of 1,623 questions across 6 task types on 29 IKEA furniture products, and evaluate 19 VLMs (2B-38B) under three alignment strategies. Our key findings: (1) assembly instruction understanding is recoverable via text, but text simultaneously degrades diagram-to-video alignment; (2) architecture family predicts alignment accuracy more strongly than parameter count; (3) video understanding remains a hard bottleneck unaffected by strategy. A three-level mechanistic analysis further reveals that diagrams and video occupy disjoint ViT subspaces, and that adding text shifts models from visual to text-driven reasoning. These results identify visual encoding as the primary target for improving cross-depiction robustness. Project page: this https URL
Abstract (translated)
二维装配图通常抽象且难以理解，这催生了对能够监控进度、检测错误并提供逐步指导的智能助手的需求。在混合现实场景中，此类系统必须通过摄像头画面识别已完成和进行中的步骤，并将其与图示指令对齐。视觉语言模型（VLMs）在此任务中展现出潜力，但面临表征鸿沟——装配图与视频帧共享的视觉特征极少。为系统评估这一差距，我们构建了IKEA-Bench基准测试，涵盖29款宜家家具产品的6类任务共1623个问题，并评估了19个（20亿至380亿参数）视觉语言模型在三种对齐策略下的表现。核心发现如下：(1) 装配指令理解可通过文本恢复，但文本同时削弱了图-视频对齐能力；(2) 架构家族比参数量更显著影响对齐精度；(3) 视频理解仍是策略无法缓解的关键瓶颈。三级机理分析进一步揭示：图示与视频占据不同的ViT子空间，且添加文本会使模型从视觉推理转向文本驱动推理。这些结果指出，视觉编码是提升跨表征鲁棒性的主要优化方向。项目页面：this https URL
URL
https://arxiv.org/abs/2604.00913
PDF
-
An Approach to Enriching Surgical Video Datasets for Fine-Grained Spatial-Temporal Understanding of Vision-Language Models
2026-04-01 11:45:28Lennart Maack, Alexander SchlaeferAbstract
Surgical video understanding is a crucial prerequisite for advancing Computer-Assisted Surgery. While vision-language models (VLMs) have recently been applied to the surgical domain, existing surgical vision-language datasets lack in capturing and evaluating complex, interleaved spatial-temporal dynamics. Creating large scale datasets that accurately represent fine-grained spatial-temporal relationships in surgical videos is challenging due to costly manual annotations or error-prone generation using large language models. To address this gap, we introduce the SurgSTU-Pipeline, a deterministic generation pipeline featuring temporal and spatial continuity filtering to reliably create surgical datasets for fine-grained spatial-temporal multimodal understanding. Applying this pipeline to publicly available surgical datasets, we create the SurgSTU dataset, comprising 7515 video clips densely extended with 150k fine-grained spatial-temporal question-answer samples. Our comprehensive evaluation shows that while state-of-the-art generalist VLMs struggle in zero-shot settings, their spatial-temporal capabilities can be improved through in-context learning. A fine-tuned VLM on the SurgSTU training dataset achieves highest performance among all spatial-temporal tasks, validating the dataset's efficacy to improve spatial-temporal understanding of VLMs in surgical videos. Code will be made publicly available.
Abstract (translated)
手术视频理解是推进计算机辅助手术的关键前提。尽管视觉语言模型（VLMs）近期已被应用于外科领域，但现有手术视觉语言数据集在捕捉和评估复杂交织的时空动态方面存在不足。由于人工标注成本高昂，或使用大语言模型生成时易产生错误，创建能准确表征手术视频中细粒度时空关系的大规模数据集极具挑战。为填补此空白，我们提出了SurgSTU-Pipeline——一种具备时空连续性过滤功能的确定性生成流程，可可靠地创建用于细粒度时空多模态理解的手术数据集。通过将该流程应用于公开手术数据集，我们构建了SurgSTU数据集，该数据集包含7515个视频片段，并密集扩展了15万条细粒度时空问答样本。全面评估表明，尽管最先进的通用视觉语言模型在零样本设置下表现欠佳，但其时空能力可通过上下文学习得到提升。在SurgSTU训练数据集上微调的视觉语言模型在所有时空任务中达到最高性能，验证了该数据集对提升视觉语言模型手术视频时空理解能力的有效性。代码将公开提供。
URL
https://arxiv.org/abs/2604.00784
PDF
-
IWP: Token Pruning as Implicit Weight Pruning in Large Vision Language Models
2026-04-01 11:23:16Dong-Jae Lee, Sunghyun Baek, Junmo KimAbstract
Large Vision Language Models show impressive performance across image and video understanding tasks, yet their computational cost grows rapidly with the number of visual tokens. Existing token pruning methods mitigate this issue through empirical approaches while overlooking the internal mechanism of attention. In this paper, we propose a novel training free token pruning framework grounded in the dual form perspective of attention. We reformulate attention as an implicit linear layer whose weight matrix is the sum of rank 1 outer products, each generated by a single token's key value pair. Token pruning thus reduces to selecting an optimal subset of these rank 1 updates that best approximates the original dual weight matrix. Extending this perspective to standard softmax attention in LVLMs, we derive a novel metric quantifying both a token's information magnitude and information duplication. To efficiently select the subset with the proposed metric, we introduce Progressive Chunked Maximal Marginal Relevance. Extensive experiments demonstrate that our method achieves a better trade off between performance and efficiency, while providing another perspective on existing pruning approaches.
Abstract (translated)
大型视觉语言模型在图像与视频理解任务中展现出卓越性能，但其计算成本随视觉token数量急剧增加。现有token剪枝方法多依赖经验性策略，却忽视了注意力的内在机制。本文提出一种基于注意力对偶形式视角的新型无训练token剪枝框架。我们将注意力重构为一个隐式线性层，其权重矩阵由多个秩1外积之和构成，每个外积由单个token的键值对生成。因此，token剪枝问题转化为选择最优子集，使这些秩1更新能最佳逼近原始对偶权重矩阵。将该视角扩展至LVLM的标准softmax注意力，我们提出一种同时量化token信息量及信息冗余的新指标。为高效筛选指标定义的子集，我们设计了渐进分块最大边际相关性算法。大量实验表明，该方法在性能与效率间取得更优平衡，并为现有剪枝方法提供了新的理论视角。
URL
https://arxiv.org/abs/2604.00757
PDF
-
CL-VISTA: Benchmarking Continual Learning in Video Large Language Models
2026-04-01 09:30:47Haiyang Guo, Yichen Shi, Fei Zhu, Wenzhuo Liu, Hongbo Zhao, Fanhu Zeng, Shijie Ma, Da-Han Wang, Xu-Yao ZhangAbstract
Video Large Language Models (Video-LLMs) require continual learning to adapt to non-stationary real-world data. However, existing benchmarks fall short of evaluating modern foundation models: many still rely on models without large-scale pre-training, and prevailing benchmarks typically partition a single dataset into sub-tasks, resulting in high task redundancy and negligible forgetting on pre-trained Video-LLMs. To address these limitations, we propose CL-VISTA, a benchmark tailored for continual video understanding of Video-LLMs. By curating 8 diverse tasks spanning perception, understanding, and reasoning, CL-VISTA induces substantial distribution shifts that effectively expose catastrophic forgetting. To systematically assess CL methods, we establish a comprehensive evaluation framework comprising 6 distinct protocols across 3 critical dimensions: performance, computational efficiency, and memory footprint. Notably, the performance dimension incorporates a general video understanding assessment to assess whether CL methods genuinely enhance foundational intelligence or merely induce task-specific overfitting. Extensive benchmarking of 10 mainstream CL methods reveals a fundamental trade-off: no single approach achieves universal superiority across all dimensions. Methods that successfully mitigate catastrophic forgetting tend to compromise generalization or incur prohibitive computational and memory overheads. We hope CL-VISTA provides critical insights for advancing continual learning in multimodal foundation models.
Abstract (translated)
视频大语言模型需要持续学习以适应非平稳的真实世界数据。然而，现有基准测试在评估现代基础模型时存在不足：许多仍依赖未经大规模预训练的模型，且主流基准通常将单一数据集划分为多个子任务，导致任务冗余度高，对预训练的视频大语言模型几乎不产生遗忘效应。为应对这些局限，我们提出了CL-VISTA——一个专为视频大语言模型持续视频理解设计的基准测试。通过精心选取涵盖感知、理解与推理三大维度的8项多样化任务，CL-VISTA能引发显著的数据分布偏移，从而有效暴露灾难性遗忘现象。为系统评估持续学习方法，我们建立了包含性能、计算效率与内存占用三个关键维度的6种评估协议的综合框架。其中性能维度特别引入通用视频理解评估，以判断持续学习方法是否真正提升基础智能，还是仅导致任务特定过拟合。对10种主流持续学习方法的大规模基准测试揭示了一个根本性权衡：没有任何单一方法能在所有维度实现全面优势。成功缓解灾难性遗忘的方法往往以牺牲泛化能力或产生高昂的计算与内存开销为代价。我们希望CL-VISTA能为推动多模态基础模型的持续学习研究提供关键洞见。
URL
https://arxiv.org/abs/2604.00677
PDF
-
Scaling Video Pretraining for Surgical Foundation Models
2026-03-31 16:31:25Sicheng Lu, Zikai Xiao, Jianhui Wei, Danyu Sun, Qi Lu, Keli Hu, Yang Feng, Jian Wu, Zongxin Yang, Zuozhu LiuAbstract
Surgical video understanding is essential for computer-assisted interventions, yet existing surgical foundation models remain constrained by limited data scale, procedural diversity, and inconsistent evaluation, often lacking a reproducible training pipeline. We propose SurgRec, a scalable and reproducible pretraining recipe for surgical video understanding, instantiated with two variants: SurgRec-MAE and SurgRec-JEPA. We curate a large multi-source corpus of 10,535 videos and 214.5M frames spanning endoscopy, laparoscopy, cataract, and robotic surgery. Building on this corpus, we develop a unified pretraining pipeline with balanced sampling and standardize a reproducible benchmark across 16 downstream datasets and four clinical domains with consistent data splits. Across extensive comparisons against SSL baselines and vision-language models, SurgRec consistently achieves superior performance across downstream datasets. In contrast, VLMs prove unreliable for fine-grained temporal recognition, exhibiting both performance gaps and sensitivity to prompt phrasing. Our work provides a reproducible, scalable foundation for the community to build more general surgical video models. All code, models, and data will be publicly released.
Abstract (translated)
手术视频理解对于计算机辅助干预至关重要，然而现有手术基础模型仍受限于数据规模有限、流程多样性不足以及评估标准不统一等问题，往往缺乏可复现的训练流程。本文提出SurgRec，一种可扩展且可复现的手术视频理解预训练方案，具体实现两个变体：SurgRec-MAE与SurgRec-JEPA。我们构建了一个涵盖内窥镜、腹腔镜、白内障及机器人手术的大型多源语料库，包含10,535个视频与2.145亿帧图像。基于该语料库，我们开发了统一的预训练流程，采用平衡采样策略，并在涵盖16个下游数据集与四个临床领域的标准化可复现基准上统一数据划分。在与自监督学习基线及视觉语言模型的广泛对比中，SurgRec在所有下游数据集上均表现优异。相比之下，视觉语言模型在细粒度时序识别任务中可靠性不足，既存在性能差距，又对提示词表述敏感。本研究为社区构建更通用手术视频模型提供了可复现、可扩展的基础框架。所有代码、模型与数据将公开发布。
URL
https://arxiv.org/abs/2603.29966
PDF
-
Video-Oasis: Rethinking Evaluation of Video Understanding
2026-03-31 11:37:42Geuntaek Lim, Minho Shim, Sungjune Park, Jaeyun Lee, Inwoong Lee, Taeoh Kim, Dongyoon Wee, Yukyung ChoiAbstract
The inherent complexity of video understanding makes it difficult to attribute whether performance gains stem from visual perception, linguistic reasoning, or knowledge priors. While many benchmarks have emerged to assess high-level reasoning, the essential criteria that constitute video understanding remain largely overlooked. Instead of introducing yet another benchmark, we take a step back to re-examine the current landscape of video understanding. In this work, we provide Video-Oasis, a sustainable diagnostic suite designed to systematically evaluate existing evaluations and distill spatio-temporal challenges for video understanding. Our analysis reveals two critical findings: (1) 54% of existing benchmark samples are solvable without visual input or temporal context, and (2) on the remaining samples, state-of-the-art models exhibit performance barely exceeding random guessing. To bridge this gap, we investigate which algorithmic design choices contribute to robust video understanding, providing practical guidelines for future research. We hope our work serves as a standard guideline for benchmark construction and the rigorous evaluation of architecture development. Code is available at this https URL.
Abstract (translated)
视频理解固有的复杂性使得难以归因性能提升究竟源于视觉感知、语言推理还是知识先验。尽管已涌现出许多评估高层推理的基准测试，但构成视频理解的核心标准却 largely 被忽视。我们并未推出另一个新基准，而是退后一步重新审视视频理解的现状。本研究提供 Video-Oasis——一个可持续的诊断套件，旨在系统评估现有评测并提炼视频理解的时空挑战。我们的分析揭示两个关键发现：(1) 现有基准样本中 54% 可在无视觉输入或时序上下文的情况下解决；(2) 在剩余样本上，最先进模型的性能仅略高于随机猜测。为弥合这一差距，我们探究哪些算法设计选择有助于实现鲁棒的视频理解，并为未来研究提供实用指南。我们希望本研究能为基准构建和架构开发的严格评估提供标准指南。代码已公开于此 https URL。
URL
https://arxiv.org/abs/2603.29616
PDF
-
VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference
2026-03-31 09:38:31Anmin Liu, Ruixuan Yang, Huiqiang Jiang, Bin Lin, Minmin Sun, Yong Li, Chen Zhang, Tao XieAbstract
Long-context video understanding and generation pose a significant computational challenge for Transformer-based video models due to the quadratic complexity of self-attention. While existing sparse attention methods employ coarse-grained patterns to improve efficiency, they typically incur redundant computation and suboptimal performance. To address this issue, in this paper, we propose \textbf{VecAttention}, a novel framework of vector-wise sparse attention that achieves superior accuracy-efficiency trade-offs for video models. We observe that video attention maps exhibit a strong vertical-vector sparse pattern, and further demonstrate that this vertical-vector pattern offers consistently better accuracy-sparsity trade-offs compared with existing coarse-grained sparse patterns. Based on this observation, VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection that minimizes memory access overhead and an optimized kernel of vector sparse attention. Comprehensive evaluations on video understanding (VideoMME, LongVideoBench, and VCRBench) and generation (VBench) tasks show that VecAttention delivers a 2.65$\times$ speedup over full attention and a 1.83$\times$ speedup over state-of-the-art sparse attention methods, with comparable accuracy to full attention. Our code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.29494
PDF
-
Scaling the Long Video Understanding of Multimodal Large Language Models via Visual Memory Mechanism
2026-03-31 04:23:33Tao Chen, Kun Zhang, Qiong Wu, Xiao Chen, Chao Chang, Xiaoshuai Sun, Yiyi Zhou, Rongrong JiAbstract
Long video understanding is a key challenge that plagues the advancement of \emph{Multimodal Large language Models} (MLLMs). In this paper, we study this problem from the perspective of visual memory mechanism, and proposed a novel and training-free approach, termed \emph{Flexible Memory} (\textbf{FlexMem}). In principle, FlexMem aims to mimic human behavior of video watching, \emph{i.e.}, continually watching video content and recalling the most relevant memory fragments to answer the question. In this way, FlexMem can help MLLMs achieve video understanding of infinite lengths, unlike previous methods that process all video information at once and have input upper-limit. Concretely, FlexMem first consider the visual KV caches as the memory sources, and realize the effective memory transfer and writing via a dual-pathway compression design. Afterwards, FlexMem also explores different memory reading strategies for the diverse video understanding tasks, including the popular streaming one. To validate FlexMem, we apply it to two popular video-MLLMs, and conduct extensive experiments on five long video and one streaming video task. The experimental results show that on \textbf{a single 3090 GPU}, our FlexMem can achieve obvious improvements than existing efficient video understanding methods and process more than \textbf{1k frames}, which also helps the base MLLMs achieve comparable or even better performance than SOTA MLLMs on some benchmarks, \emph{e.g.} , GPT-4o and Gemini-1.5 Pro.
Abstract (translated)
URL
https://arxiv.org/abs/2603.29252
PDF
-
AdaptToken: Entropy-based Adaptive Token Selection for MLLM Long Video Understanding
2026-03-30 17:14:15Haozhe Qi, Kevin Qu, Mahdi Rad, Rui Wang, Alexander Mathis, Marc PollefeysAbstract
Long video understanding remains challenging for Multi-modal Large Language Models (MLLMs) due to high memory costs and context-length limits. Prior approaches mitigate this by scoring and selecting frames/tokens within short clips, but they lack a principled mechanism to (i) compare relevance across distant video clips and (ii) stop processing once sufficient evidence has been gathered. We propose AdaptToken, a training-free framework that turns an MLLM's self-uncertainty into a global control signal for long-video token selection. AdaptToken splits a video into groups, extracts cross-modal attention to rank tokens within each group, and uses the model's response entropy to estimate each group's prompt relevance. This entropy signal enables a global token budget allocation across groups and further supports early stopping (AdaptToken-Lite), skipping the remaining groups when the model becomes sufficiently certain. Across four long-video benchmarks (VideoMME, LongVideoBench, LVBench, and MLVU) and multiple base MLLMs (7B-72B), AdaptToken consistently improves accuracy (e.g., +6.7 on average over Qwen2.5-VL 7B) and continues to benefit from extremely long inputs (up to 10K frames), while AdaptToken-Lite reduces inference time by about half with comparable performance. Project page: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.28696
PDF
-
Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Models
2026-03-30 05:11:45Nanxi Li, Xiang Wang, Yuanjie Chen, Haode Zhang, Hong Li, Yong-Lu LiAbstract
While Multimodal Large Language Models (MLLMs) have demonstrated impressive capabilities in image and video understanding, their ability to comprehend the physical world has become an increasingly important research focus. Despite their improvements, current MLLMs struggle significantly with high-level physics reasoning. In this work, we investigate the first step of physical reasoning, i.e., intuitive physics understanding, revealing substantial limitations in understanding the dynamics of continuum objects. To isolate and evaluate this specific capability, we introduce two fundamental benchmark tasks: Next Frame Selection (NFS) and Temporal Coherence Verification (TCV). Our experiments demonstrate that even state-of-the-art MLLMs perform poorly on these foundational tasks. To address this limitation, we propose Scene Dynamic Field (SDF), a concise approach that leverages physics simulators within a multi-task fine-tuning framework. SDF substantially improves performance, achieving up to 20.7% gains on fluid tasks while showing strong generalization to unseen physical domains. This work not only highlights a critical gap in current MLLMs but also presents a promising cost-efficient approach for developing more physically grounded MLLMs. Our code and data are available at this https URL.
Abstract (translated)
尽管多模态大语言模型（MLLMs）在图像与视频理解方面已展现出卓越能力，但其对物理世界的理解正日益成为关键研究焦点。尽管性能持续提升，当前MLLMs在高阶物理推理方面仍存在显著不足。本研究聚焦物理推理的首个环节——直觉物理理解，揭示了模型在连续体物体动态认知上的重大局限。为独立评估该能力，我们引入两项基础基准任务：下一帧选择（NFS）与时序一致性验证（TCV）。实验表明，即便是最先进的MLLMs在这些基础任务上表现仍欠佳。针对此局限，我们提出场景动态场（SDF）方法——一种基于多任务微调框架、融合物理模拟器的简洁方案。SDF显著提升性能，在流体任务上最高实现20.7%的增益，并对未见物理领域展现出强泛化能力。本研究不仅揭示了当前MLLMs的关键缺陷，更为开发更具物理基础性的模型提供了高性价比的可行路径。代码与数据已公开于此链接。
URL
https://arxiv.org/abs/2604.03302
PDF
