Video_Caption
-
CauCLIP: Bridging the Sim-to-Real Gap in Surgical Video Understanding via Causality-Inspired Vision-Language Modeling
2026-02-06 11:23:17Yuxin He, An Li, Cheng XueAbstract
Surgical phase recognition is a critical component for context-aware decision support in intelligent operating rooms, yet training robust models is hindered by limited annotated clinical videos and large domain gaps between synthetic and real surgical data. To address this, we propose CauCLIP, a causality-inspired vision-language framework that leverages CLIP to learn domain-invariant representations for surgical phase recognition without access to target domain data. Our approach integrates a frequency-based augmentation strategy to perturb domain-specific attributes while preserving semantic structures, and a causal suppression loss that mitigates non-causal biases and reinforces causal surgical features. These components are combined in a unified training framework that enables the model to focus on stable causal factors underlying surgical workflows. Experiments on the SurgVisDom hard adaptation benchmark demonstrate that our method substantially outperforms all competing approaches, highlighting the effectiveness of causality-guided vision-language models for domain-generalizable surgical video understanding.
Abstract (translated)
手术阶段识别是智能手术室中上下文感知决策支持的关键组成部分，然而由于临床视频的标注数据有限以及合成和真实外科数据之间的领域差异较大，训练稳健模型面临挑战。为解决这些问题，我们提出了一种受因果关系启发的视觉-语言框架CauCLIP，该框架利用CLIP（对比语言-图像预训练）来学习无需访问目标领域数据即可进行手术阶段识别的领域不变表示。 我们的方法结合了一个基于频率的数据增强策略，可以扰动特定领域的属性同时保留语义结构，并引入了一种因果抑制损失函数以减轻非因果偏见并强化因果外科特征。这些组件被整合进一个统一的训练框架中，使模型能够专注于手术流程背后稳定的因果因素。 在SurgVisDom硬适应基准上的实验表明，我们的方法显著优于所有竞争的方法，这凸显了受因果关系指导的视觉-语言模型在跨领域手术视频理解中的有效性。
URL
https://arxiv.org/abs/2602.06619
PDF
-
UniSurg: A Video-Native Foundation Model for Universal Understanding of Surgical Videos
2026-02-05 13:18:33Jinlin Wu, Felix Holm, Chuxi Chen, An Wang, Yaxin Hu, Xiaofan Ye, Zelin Zang, Miao Xu, Lihua Zhou, Huai Liao, Danny T. M. Chan, Ming Feng, Wai S. Poon, Hongliang Ren, Dong Yi, Nassir Navab, Gaofeng Meng, Jiebo Luo, Hongbin Liu, Zhen LeiAbstract
While foundation models have advanced surgical video analysis, current approaches rely predominantly on pixel-level reconstruction objectives that waste model capacity on low-level visual details - such as smoke, specular reflections, and fluid motion - rather than semantic structures essential for surgical understanding. We present UniSurg, a video-native foundation model that shifts the learning paradigm from pixel-level reconstruction to latent motion prediction. Built on the Video Joint Embedding Predictive Architecture (V-JEPA), UniSurg introduces three key technical innovations tailored to surgical videos: 1) motion-guided latent prediction to prioritize semantically meaningful regions, 2) spatiotemporal affinity self-distillation to enforce relational consistency, and 3) feature diversity regularization to prevent representation collapse in texture-sparse surgical scenes. To enable large-scale pretraining, we curate UniSurg-15M, the largest surgical video dataset to date, comprising 3,658 hours of video from 50 sources across 13 anatomical regions. Extensive experiments across 17 benchmarks demonstrate that UniSurg significantly outperforms state-of-the-art methods on surgical workflow recognition (+14.6% F1 on EgoSurgery, +10.3% on PitVis), action triplet recognition (39.54% mAP-IVT on CholecT50), skill assessment, polyp segmentation, and depth estimation. These results establish UniSurg as a new standard for universal, motion-oriented surgical video understanding.
Abstract (translated)
尽管基础模型在手术视频分析方面取得了进展，但目前的方法主要依赖于像素级别的重构目标，这导致模型容量被浪费在低级视觉细节（如烟雾、镜面反射和液体流动）上，而不是对理解手术至关重要的语义结构。我们介绍了UniSurg，这是一种基于视频的基础模型，它将学习范式从像素级别重建转向潜在的运动预测。UniSurg建立在视频联合嵌入预测架构(V-JEPA)的基础上，并为手术视频引入了三项关键技术创新： 1. **以运动引导潜在预测**：优先考虑语义上有意义的区域。 2. **时空亲和自我蒸馏**：强制执行关系一致性。 3. **特征多样性正则化**：防止在纹理稀疏的手术场景中出现表示崩溃。 为了实现大规模预训练，我们整理了UniSurg-15M，这是迄今为止最大的手术视频数据集，包含来自50个来源、跨越13个解剖区域的3,658小时视频。通过17项基准测试的大规模实验表明，UniSurg在手术流程识别（EgoSurgery上的F1分数提高14.6%，PitVis上提高10.3%）、动作三元组识别（CholecT50上的mAP-IVT分数为39.54%）、技能评估、息肉分割和深度估计方面，均显著超越了现有最佳方法。这些结果确立了UniSurg作为通用且以运动为导向的手术视频理解新标准的地位。
URL
https://arxiv.org/abs/2602.05638
PDF
-
OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
2026-02-04 17:51:05Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, Yuanxing Zhang, Jiaheng Liu, Qiang Liu, Pengfei Wan, Liang WangAbstract
Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks.
Abstract (translated)
跨模态大型语言模型（Omni-LLMs）在音频视频理解任务中展示了强大的能力，但它们依赖于长的多模态令牌序列导致了显著的计算开销。尽管存在这一挑战，为Omni-LLM设计的令牌压缩方法仍然有限。为了弥合这一差距，我们提出了OmniSIFT（跨模态时空信息细粒度令牌压缩），这是一种针对Omni-LLMs定制的模态非对称令牌压缩框架。具体而言，OmniSIFT采用了一种两阶段压缩策略：(i) 一个时空视频修剪模块，移除来自帧内结构和帧间重叠的视频冗余；(ii) 一个视觉引导的音频选择模块，过滤音频令牌。整个框架通过可微分直通估计器进行端到端优化。 在五个代表性基准上的广泛实验展示了OmniSIFT的有效性和鲁棒性。值得注意的是，对于Qwen2.5-Omni-7B模型，虽然仅引入了4.85M参数，但OmniSIFT保持了比无训练基线（如OmniZip）更低的延迟。在仅有原始令牌上下文的25%的情况下，OmniSIFT始终超越所有压缩基线，并且在若干任务上甚至超过了完整令牌模型的表现。
URL
https://arxiv.org/abs/2602.04804
PDF
-
VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents
2026-02-04 04:39:46Feng Wang, Yichun Shi, Ceyuan Yang, Qiushan Guo, Jingxiang Sun, Alan Yuille, Peng WangAbstract
This work presents VTok, a unified video tokenization framework that can be used for both generation and understanding tasks. Unlike the leading vision-language systems that tokenize videos through a naive frame-sampling strategy, we propose to decouple the spatial and temporal representations of videos by retaining the spatial features of a single key frame while encoding each subsequent frame into a single residual token, achieving compact yet expressive video tokenization. Our experiments suggest that VTok effectively reduces the complexity of video representation from the product of frame count and per-frame token count to their sum, while the residual tokens sufficiently capture viewpoint and motion changes relative to the key frame. Extensive evaluations demonstrate the efficacy and efficiency of VTok: it achieves notably higher performance on a range of video understanding and text-to-video generation benchmarks compared with baselines using naive tokenization, all with shorter token sequences per video (e.g., 3.4% higher accuracy on our TV-Align benchmark and 1.9% higher VBench score). Remarkably, VTok produces more coherent motion and stronger guidance following in text-to-video generation, owing to its more consistent temporal encoding. We hope VTok can serve as a standardized video tokenization paradigm for future research in video understanding and generation.
Abstract (translated)
这项工作提出了一种名为VTok的统一视频标记化框架，该框架可用于生成和理解任务。与现有的主流视觉-语言系统采用简单的帧采样策略对视频进行标记化不同，我们提议通过保留单个关键帧的空间特征，并将每个后续帧编码为一个残差令牌来解耦视频的空间和时间表示，从而实现紧凑且表达丰富的视频标记化。实验表明，VTok有效减少了视频表征的复杂度，从每秒帧数与每帧令牌数量的乘积降低到它们的总和，同时残差令牌充分捕捉了相对于关键帧的角度变化和运动变化。 广泛的评估证明了VTok的有效性和效率：相较于使用简单标记化的基线模型，在各种视频理解和文本生成视频基准测试中取得了显著更高的性能（例如，在我们的TV-Align基准上准确性提高了3.4%，在VBench得分上提高了1.9%）。值得注意的是，由于其一致的时间编码，VTok在文本到视频的生成过程中产生了更为连贯的动作和更强的指导。我们希望VTok能够作为未来研究中标准化的视频标记化范式，在视频理解和生成方面发挥作用。
URL
https://arxiv.org/abs/2602.04202
PDF
-
VideoBrain: Learning Adaptive Frame Sampling for Long Video Understanding
2026-02-04 00:08:35Junbo Zou, Ziheng Huang, Shengjie Zhang, Liwen Zhang, Weining ShenAbstract
Long-form video understanding remains challenging for Vision-Language Models (VLMs) due to the inherent tension between computational constraints and the need to capture information distributed across thousands of frames. Existing approaches either sample frames uniformly (risking information loss) or select keyframes in a single pass (with no recovery from poor choices). We propose VideoBrain, an end-to-end framework that enables VLMs to adaptively acquire visual information through learned sampling policies. Our approach features dual complementary agents: a CLIP-based agent for semantic retrieval across the video and a Uniform agent for dense temporal sampling within intervals. Unlike prior agent-based methods that rely on text-only LLMs orchestrating visual tools, our VLM directly perceives frames and reasons about information sufficiency. To prevent models from invoking agents indiscriminately to maximize rewards, we introduce a behavior-aware reward function coupled with a data classification pipeline that teaches the model when agent invocation is genuinely beneficial. Experiments on four long video benchmarks demonstrate that VideoBrain achieves +3.5% to +9.0% improvement over the baseline while using 30-40% fewer frames, with strong cross-dataset generalization to short video benchmarks.
Abstract (translated)
长视频理解对视觉语言模型（VLMs）来说仍是一个挑战，这主要是由于计算限制与捕捉分布在数千帧中信息的需求之间的内在矛盾。现有的方法要么是均匀采样帧（有丢失信息的风险），要么是在一次通过时选择关键帧（无法从糟糕的选择中恢复）。我们提出了VideoBrain，这是一种端到端框架，它使VLMs能够通过学习的采样策略自适应地获取视觉信息。我们的方法包含两个互补的代理：一个基于CLIP的代理用于视频中的语义检索和一个均匀代理用于在间隔内进行密集的时间采样。与依赖于仅使用文本的大规模语言模型（LLM）来指挥视觉工具的先前代理基线方法不同，我们的VLM直接感知帧并判断信息是否充足。 为了防止模型无差别地调用代理以最大化奖励，我们引入了一个行为感知奖励函数，并结合数据分类管道，教导模型何时调用代理是真正有益的。在四个长视频基准测试上的实验表明，VideoBrain相比于基线方法，在使用30-40%更少帧的情况下实现了+3.5%到+9.0%的改进，并且对短视频基准有强大的跨数据集泛化能力。
URL
https://arxiv.org/abs/2602.04094
PDF
-
KTV: Keyframes and Key Tokens Selection for Efficient Training-Free Video LLMs
2026-02-03 15:08:30Baiyang Song, Jun Peng, Yuxin Zhang, Guangyao Chen, Feidiao Yang, Jianyuan GuoAbstract
Training-free video understanding leverages the strong image comprehension capabilities of pre-trained vision language models (VLMs) by treating a video as a sequence of static frames, thus obviating the need for costly video-specific training. However, this paradigm often suffers from severe visual redundancy and high computational overhead, especially when processing long videos. Crucially, existing keyframe selection strategies, especially those based on CLIP similarity, are prone to biases and may inadvertently overlook critical frames, resulting in suboptimal video comprehension. To address these significant challenges, we propose \textbf{KTV}, a novel two-stage framework for efficient and effective training-free video understanding. In the first stage, KTV performs question-agnostic keyframe selection by clustering frame-level visual features, yielding a compact, diverse, and representative subset of frames that mitigates temporal redundancy. In the second stage, KTV applies key visual token selection, pruning redundant or less informative tokens from each selected keyframe based on token importance and redundancy, which significantly reduces the number of tokens fed into the LLM. Extensive experiments on the Multiple-Choice VideoQA task demonstrate that KTV outperforms state-of-the-art training-free baselines while using significantly fewer visual tokens, \emph{e.g.}, only 504 visual tokens for a 60-min video with 10800 frames, achieving $44.8\%$ accuracy on the MLVU-Test benchmark. In particular, KTV also exceeds several training-based approaches on certain benchmarks.
Abstract (translated)
无训练视频理解方法利用预训练的视觉语言模型（VLM）的强大图像理解能力，将视频视为一系列静态帧序列，从而避免了昂贵的特定于视频的训练需求。然而，这种范式在处理长视频时往往遭受严重的视觉冗余和高昂的计算开销问题。特别是基于CLIP相似性的现有关键帧选择策略容易产生偏差，并可能无意中忽略关键帧，导致次优的视频理解结果。 为了解决这些重大挑战，我们提出了一种名为KTV的新颖两阶段框架，用于高效的无训练视频理解。在第一阶段，KTV通过聚类帧级别的视觉特征执行与问题无关的关键帧选择，从而产生一个精简、多样且具代表性的关键帧子集，有效减少了时间冗余。在第二阶段，KTV应用关键视觉标记选择方法，根据每个选定关键帧中令牌的重要性及冗余度去除多余的或信息量较小的令牌，显著减少输入LLM（大型语言模型）中的视觉令牌数量。 广泛的实验显示，在多项选择视频问答任务上，KTV的表现超越了最先进的无训练基准，并且使用了显著较少的视觉令牌。例如，对于一个包含10800帧的60分钟视频，仅需504个视觉令牌即可在MLVU-Test基准测试中达到44.8%的准确率。此外，在某些基准上，KTV的表现甚至超过了几个基于训练的方法。 这个描述突出了KTV框架如何通过两个阶段有效减少冗余，并提高无训练视频理解模型的效果和效率。
URL
https://arxiv.org/abs/2602.03615
PDF
-
SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM
2026-02-03 14:39:16Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, Li ZhangAbstract
Large language models (LLMs) have demonstrated exceptional capabilities in text understanding, which has paved the way for their expansion into video LLMs (Vid-LLMs) to analyze video data. However, current Vid-LLMs struggle to simultaneously retain high-quality frame-level semantic information (i.e., a sufficient number of tokens per frame) and comprehensive video-level temporal information (i.e., an adequate number of sampled frames per video). This limitation hinders the advancement of Vid-LLMs towards fine-grained video understanding. To address this issue, we introduce the SlowFocus mechanism, which significantly enhances the equivalent sampling frequency without compromising the quality of frame-level visual tokens. SlowFocus begins by identifying the query-related temporal segment based on the posed question, then performs dense sampling on this segment to extract local high-frequency features. A multi-frequency mixing attention module is further leveraged to aggregate these local high-frequency details with global low-frequency contexts for enhanced temporal comprehension. Additionally, to tailor Vid-LLMs to this innovative mechanism, we introduce a set of training strategies aimed at bolstering both temporal grounding and detailed temporal reasoning capabilities. Furthermore, we establish FineAction-CGR, a benchmark specifically devised to assess the ability of Vid-LLMs to process fine-grained temporal understanding tasks. Comprehensive experiments demonstrate the superiority of our mechanism across both existing public video understanding benchmarks and our proposed FineAction-CGR.
Abstract (translated)
大型语言模型（LLM）已经在文本理解方面表现出色，这为它们扩展到视频LLM（Vid-LLM），以分析视频数据铺平了道路。然而，当前的Vid-LLM在同时保留高质量的帧级语义信息（即每帧有足够的token数量）和全面的视频级时间信息（即每个视频采样的足够多的帧数）方面遇到困难。这种限制阻碍了Vid-LLMs向细粒度视频理解的进步。为了解决这个问题，我们引入了SlowFocus机制，该机制显著提高了等效采样频率，同时不损害帧级视觉token的质量。SlowFocus首先根据提出的问题确定查询相关的时间段，然后在此时间段上进行密集采样以提取局部高频特征。进一步利用多频混合注意模块将这些局部高频细节与全局低频上下文聚合起来，从而增强时间理解能力。此外，为了使Vid-LLM适应这种创新机制，我们还引入了一系列旨在加强时间和详细时间推理能力的训练策略。另外，我们建立了一个基准测试FineAction-CGR，专门用于评估Vid-LLMs处理细粒度时间理解任务的能力。全面的实验表明，在现有的公开视频理解基准和我们提出的FineAction-CGR上，我们的机制都表现出了优越性。
URL
https://arxiv.org/abs/2602.03589
PDF
-
LongVPO: From Anchored Cues to Self-Reasoning for Long-Form Video Preference Optimization
2026-02-02 17:03:37Zhenpeng Huang, Jiaqi Li, Zihan Jia, Xinhao Li, Desen Meng, Lingxue Song, Xi Chen, Liang Li, Limin WangAbstract
We present LongVPO, a novel two-stage Direct Preference Optimization framework that enables short-context vision-language models to robustly understand ultra-long videos without any long-video annotations. In Stage 1, we synthesize preference triples by anchoring questions to individual short clips, interleaving them with distractors, and applying visual-similarity and question-specificity filtering to mitigate positional bias and ensure unambiguous supervision. We also approximate the reference model's scoring over long contexts by evaluating only the anchor clip, reducing computational overhead. In Stage 2, we employ a recursive captioning pipeline on long videos to generate scene-level metadata, then use a large language model to craft multi-segment reasoning queries and dispreferred responses, aligning the model's preferences through multi-segment reasoning tasks. With only 16K synthetic examples and no costly human labels, LongVPO outperforms the state-of-the-art open-source models on multiple long-video benchmarks, while maintaining strong short-video performance (e.g., on MVBench), offering a scalable paradigm for efficient long-form video understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2602.02341
PDF
-
FreshMem: Brain-Inspired Frequency-Space Hybrid Memory for Streaming Video Understanding
2026-02-02 05:52:11Kangcong Li, Peng Ye, Lin Zhang, Chao Wang, Huafeng Qin, Tao ChenAbstract
Transitioning Multimodal Large Language Models (MLLMs) from offline to online streaming video understanding is essential for continuous perception. However, existing methods lack flexible adaptivity, leading to irreversible detail loss and context fragmentation. To resolve this, we propose FreshMem, a Frequency-Space Hybrid Memory network inspired by the brain's logarithmic perception and memory consolidation. FreshMem reconciles short-term fidelity with long-term coherence through two synergistic modules: Multi-scale Frequency Memory (MFM), which projects overflowing frames into representative frequency coefficients, complemented by residual details to reconstruct a global historical "gist"; and Space Thumbnail Memory (STM), which discretizes the continuous stream into episodic clusters by employing an adaptive compression strategy to distill them into high-density space thumbnails. Extensive experiments show that FreshMem significantly boosts the Qwen2-VL baseline, yielding gains of 5.20%, 4.52%, and 2.34% on StreamingBench, OV-Bench, and OVO-Bench, respectively. As a training-free solution, FreshMem outperforms several fully fine-tuned methods, offering a highly efficient paradigm for long-horizon streaming video understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01683
PDF
-
Contribution-aware Token Compression for Efficient Video Understanding via Reinforcement Learning
2026-02-02 05:09:48Yinchao Ma, Qiang Zhou, Zhibin Wang, Xianing Chen, Hanqing Yang, Jun Song, Bo ZhengAbstract
Video large language models have demonstrated remarkable capabilities in video understanding tasks. However, the redundancy of video tokens introduces significant computational overhead during inference, limiting their practical deployment. Many compression algorithms are proposed to prioritize retaining features with the highest attention scores to minimize perturbations in attention computations. However, the correlation between attention scores and their actual contribution to correct answers remains ambiguous. To address the above limitation, we propose a novel \textbf{C}ontribution-\textbf{a}ware token \textbf{Co}mpression algorithm for \textbf{VID}eo understanding (\textbf{CaCoVID}) that explicitly optimizes the token selection policy based on the contribution of tokens to correct predictions. First, we introduce a reinforcement learning-based framework that optimizes a policy network to select video token combinations with the greatest contribution to correct predictions. This paradigm shifts the focus from passive token preservation to active discovery of optimal compressed token combinations. Secondly, we propose a combinatorial policy optimization algorithm with online combination space sampling, which dramatically reduces the exploration space for video token combinations and accelerates the convergence speed of policy optimization. Extensive experiments on diverse video understanding benchmarks demonstrate the effectiveness of CaCoVID. Codes will be released.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01649
PDF
-
Exposing and Defending the Achilles' Heel of Video Mixture-of-Experts
2026-02-01 18:12:43Songping Wang, Qinglong Liu, Yueming Lyu, Ning Li, Ziwen He, Caifeng ShanAbstract
Mixture-of-Experts (MoE) has demonstrated strong performance in video understanding tasks, yet its adversarial robustness remains underexplored. Existing attack methods often treat MoE as a unified architecture, overlooking the independent and collaborative weaknesses of key components such as routers and expert modules. To fill this gap, we propose Temporal Lipschitz-Guided Attacks (TLGA) to thoroughly investigate component-level vulnerabilities in video MoE models. We first design attacks on the router, revealing its independent weaknesses. Building on this, we introduce Joint Temporal Lipschitz-Guided Attacks (J-TLGA), which collaboratively perturb both routers and experts. This joint attack significantly amplifies adversarial effects and exposes the Achilles' Heel (collaborative weaknesses) of the MoE architecture. Based on these insights, we further propose Joint Temporal Lipschitz Adversarial Training (J-TLAT). J-TLAT performs joint training to further defend against collaborative weaknesses, enhancing component-wise robustness. Our framework is plug-and-play and reduces inference cost by more than 60% compared with dense models. It consistently enhances adversarial robustness across diverse datasets and architectures, effectively mitigating both the independent and collaborative weaknesses of MoE.
Abstract (translated)
URL
https://arxiv.org/abs/2602.01369
PDF
-
Video Understanding: Through A Temporal Lens
2026-01-31 12:01:09Thong Thanh NguyenAbstract
This thesis explores the central question of how to leverage temporal relations among video elements to advance video understanding. Addressing the limitations of existing methods, the work presents a five-fold contribution: (1) an automatic annotation framework that utilizes large vision-language models and a noise-robust contrastive learning objective with a subtractive angular margin; (2) a parameter-efficient fine-tuning strategy using "recurrent adapters" to capture temporal dynamics in low-data regimes; (3) the integration of State Space Layers (SSL) for efficient long-form video modeling, supported by the introduction of two new long-term benchmarks for egocentric and feature-length content; (4) a novel contrastive learning framework designed to explicitly model fine-grained relations between motions and video moments; and (5) a comprehensive empirical study on Large Vision-Language Models (LVLMs) that identifies the visual-language interface as a bottleneck for temporal reasoning, leading to a new "temporal-oriented recipe" for upscaled video understanding. Collectively, these contributions demonstrate that explicit temporal modeling significantly enhances a model's ability to represent and reason about the fluid nature of video content.
Abstract (translated)
URL
https://arxiv.org/abs/2602.00683
PDF
-
TimeBlind: A Spatio-Temporal Compositionality Benchmark for Video LLMs
2026-01-30 20:21:46Baiqi Li, Kangyi Zhao, Ce Zhang, Chancharik Mitra, Jean de Dieu Nyandwi, Gedas BertasiusAbstract
Fine-grained spatio-temporal understanding is essential for video reasoning and embodied AI. Yet, while Multimodal Large Language Models (MLLMs) master static semantics, their grasp of temporal dynamics remains brittle. We present TimeBlind, a diagnostic benchmark for compositional spatio-temporal understanding. Inspired by cognitive science, TimeBlind categorizes fine-grained temporal understanding into three levels: recognizing atomic events, characterizing event properties, and reasoning about event interdependencies. Unlike benchmarks that conflate recognition with temporal reasoning, TimeBlind leverages a minimal-pairs paradigm: video pairs share identical static visual content but differ solely in temporal structure, utilizing complementary questions to neutralize language priors. Evaluating over 20 state-of-the-art MLLMs (e.g., GPT-5, Gemini 3 Pro) on 600 curated instances (2400 video-question pairs), reveals that the Instance Accuracy (correctly distinguishing both videos in a pair) of the best performing MLLM is only 48.2%, far below the human performance (98.2%). These results demonstrate that even frontier models rely heavily on static visual shortcuts rather than genuine temporal logic, positioning TimeBlind as a vital diagnostic tool for next-generation video understanding. Dataset and code are available at this https URL .
Abstract (translated)
URL
https://arxiv.org/abs/2602.00288
PDF
-
Structured Over Scale: Learning Spatial Reasoning from Educational Video
2026-01-30 18:20:23Bishoy Galoaa, Xiangyu Bai, Sarah OstadabbasAbstract
Vision-language models (VLMs) demonstrate impressive performance on standard video understanding benchmarks yet fail systematically on simple reasoning tasks that preschool children can solve, including counting, spatial reasoning, and compositional understanding. We hypothesize that the pedagogically-structured content of educational videos provides an ideal training signal for improving these capabilities. We introduce DoraVQA, a dataset of 5,344 question-answer pairs automatically extracted from 8 seasons of Dora the Explorer with precise timestamp alignment. Each episode follows a consistent \textit{context-question-pause-answer} structure that creates a self-contained learning environment analogous to interactive tutoring. We fine-tune both Qwen2 and Qwen3 using Group Relative Policy Optimization (GRPO), leveraging the clear correctness signals and structured reasoning traces inherent in educational content. Despite training exclusively on 38 hours of children's educational videos, our approach achieves improvements of 8-14 points on DoraVQA and state-of-the-art 86.16\% on CVBench, with strong transfer to Video-MME and NExT-QA, demonstrating effective generalization from narrow pedagogical content to broad multimodal understanding. Through cross-domain benchmarks, we show that VLMs can perform tasks that require robust reasoning learned from structured educational content, suggesting that content structure matters as much as content scale.
Abstract (translated)
URL
https://arxiv.org/abs/2601.23251
PDF
-
Video-o3: Native Interleaved Clue Seeking for Long Video Multi-Hop Reasoning
2026-01-30 17:47:30Xiangyu Zeng, Zhiqiu Zhang, Yuhan Zhu, Xinhao Li, Zikang Wang, Changlian Ma, Qingyu Zhang, Zizheng Huang, Kun Ouyang, Tianxiang Jiang, Ziang Yan, Yi Wang, Hongjie Zhang, Yali Wang, Limin WangAbstract
Existing multimodal large language models for long-video understanding predominantly rely on uniform sampling and single-turn inference, limiting their ability to identify sparse yet critical evidence amid extensive redundancy. We introduce Video-o3, a novel framework that supports iterative discovery of salient visual clues, fine-grained inspection of key segments, and adaptive termination once sufficient evidence is acquired. Technically, we address two core challenges in interleaved tool invocation. First, to mitigate attention dispersion induced by the heterogeneity of reasoning and tool-calling, we propose Task-Decoupled Attention Masking, which isolates per-step concentration while preserving shared global context. Second, to control context length growth in multi-turn interactions, we introduce a Verifiable Trajectory-Guided Reward that balances exploration coverage with reasoning efficiency. To support training at scale, we further develop a data synthesis pipeline and construct Seeker-173K, comprising 173K high-quality tool-interaction trajectories for effective supervised and reinforcement learning. Extensive experiments show that Video-o3 substantially outperforms state-of-the-art methods, achieving 72.1% accuracy on MLVU and 46.5% on Video-Holmes. These results demonstrate Video-o3's strong multi-hop evidence-seeking and reasoning capabilities, and validate the effectiveness of native tool invocation in long-video scenarios.
Abstract (translated)
URL
https://arxiv.org/abs/2601.23224
PDF
-
Mitigating Hallucinations in Video Large Language Models via Spatiotemporal-Semantic Contrastive Decoding
2026-01-30 05:16:12Yuansheng Gao, Jinman Zhao, Tong Zhang, Xingguo Xu, Han Bao, Zonghui Wang, Wenzhi ChenAbstract
Although Video Large Language Models perform remarkably well across tasks such as video understanding, question answering, and reasoning, they still suffer from the problem of hallucination, which refers to generating outputs that are inconsistent with explicit video content or factual evidence. However, existing decoding methods for mitigating video hallucinations, while considering the spatiotemporal characteristics of videos, mostly rely on heuristic designs. As a result, they fail to precisely capture the root causes of hallucinations and their fine-grained temporal and semantic correlations, leading to limited robustness and generalization in complex scenarios. To more effectively mitigate video hallucinations, we propose a novel decoding strategy termed Spatiotemporal-Semantic Contrastive Decoding. This strategy constructs negative features by deliberately disrupting the spatiotemporal consistency and semantic associations of video features, and suppresses video hallucinations through contrastive decoding against the original video features during inference. Extensive experiments demonstrate that our method not only effectively mitigates the occurrence of hallucinations, but also preserves the general video understanding and reasoning capabilities of the model.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22574
PDF
-
Understanding Multimodal Complementarity for Single-Frame Action Anticipation
2026-01-29 17:44:23Manuel Benavent-Lledo, Konstantinos Bacharidis, Konstantinos Papoutsakis, Antonis Argyros, Jose Garcia-RodriguezAbstract
Human action anticipation is commonly treated as a video understanding problem, implicitly assuming that dense temporal information is required to reason about future actions. In this work, we challenge this assumption by investigating what can be achieved when action anticipation is constrained to a single visual observation. We ask a fundamental question: how much information about the future is already encoded in a single frame, and how can it be effectively exploited? Building on our prior work on Action Anticipation at a Glimpse (AAG), we conduct a systematic investigation of single-frame action anticipation enriched with complementary sources of information. We analyze the contribution of RGB appearance, depth-based geometric cues, and semantic representations of past actions, and investigate how different multimodal fusion strategies, keyframe selection policies and past-action history sources influence anticipation performance. Guided by these findings, we consolidate the most effective design choices into AAG+, a refined single-frame anticipation framework. Despite operating on a single frame, AAG+ consistently improves upon the original AAG and achieves performance comparable to, or exceeding, that of state-of-the-art video-based methods on challenging anticipation benchmarks including IKEA-ASM, Meccano and Assembly101. Our results offer new insights into the limits and potential of single-frame action anticipation, and clarify when dense temporal modeling is necessary and when a carefully selected glimpse is sufficient.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22039
PDF
-
SONIC-O1: A Real-World Benchmark for Evaluating Multimodal Large Language Models on Audio-Video Understanding
2026-01-29 13:01:07Ahmed Y. Radwan, Christos Emmanouilidis, Hina Tabassum, Deval Pandya, Shaina RazaAbstract
Multimodal Large Language Models (MLLMs) are a major focus of recent AI research. However, most prior work focuses on static image understanding, while their ability to process sequential audio-video data remains underexplored. This gap highlights the need for a high-quality benchmark to systematically evaluate MLLM performance in a real-world setting. We introduce SONIC-O1, a comprehensive, fully human-verified benchmark spanning 13 real-world conversational domains with 4,958 annotations and demographic metadata. SONIC-O1 evaluates MLLMs on key tasks, including open-ended summarization, multiple-choice question (MCQ) answering, and temporal localization with supporting rationales (reasoning). Experiments on closed- and open-source models reveal limitations. While the performance gap in MCQ accuracy between two model families is relatively small, we observe a substantial 22.6% performance difference in temporal localization between the best performing closed-source and open-source models. Performance further degrades across demographic groups, indicating persistent disparities in model behavior. Overall, SONIC-O1 provides an open evaluation suite for temporally grounded and socially robust multimodal understanding. We release SONIC-O1 for reproducibility and research: Project page: this https URL Dataset: this https URL Github: this https URL Leaderboard: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2601.21666
PDF
-
Spava: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention
2026-01-29 09:23:13Yuxiang Huang, Mingye Li, Xu Han, Chaojun Xiao, Weilin Zhao, Ao Sun, Ziqi Yuan, Hao Zhou, Fandong Meng, Zhiyuan LiuAbstract
The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods either compress visual embeddings or apply sparse attention on a single GPU, yielding limited acceleration or degraded performance and restricting LMMs from handling longer, more complex videos. To overcome these issues, we propose Spava, a sequence-parallel framework with optimized attention that accelerates long-video inference across multiple GPUs. By distributing approximate attention, Spava reduces computation and increases parallelism, enabling efficient processing of more visual embeddings without compression and thereby improving task performance. System-level optimizations, such as load balancing and fused forward passes, further unleash the potential of Spava, delivering speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss. Code available at this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2601.21444
PDF
-
Video-KTR: Reinforcing Video Reasoning via Key Token Attribution
2026-01-27 15:02:23Ziyue Wang, Sheng Jin, Zhongrong Zuo, Jiawei Wu, Han Qiu, Qi She, Hao Zhang, Xudong JiangAbstract
Reinforcement learning (RL) has shown strong potential for enhancing reasoning in multimodal large language models, yet existing video reasoning methods often rely on coarse sequence-level rewards or single-factor token selection, neglecting fine-grained links among visual inputs, temporal dynamics, and linguistic outputs, limiting both accuracy and interpretability. We propose Video-KTR, a modality-aware policy shaping framework that performs selective, token-level RL by combining three attribution signals: (1) visual-aware tokens identified via counterfactual masking to reveal perceptual dependence; (2) temporal-aware tokens detected through frame shuffling to expose temporal sensitivity; and (3) high-entropy tokens signaling predictive uncertainty. By reinforcing only these key tokens, Video-KTR focuses learning on semantically informative, modality-sensitive content while filtering out low-value tokens. Across five challenging benchmarks, Video-KTR achieves state-of-the-art or highly competitive results, achieving 42.7\% on Video-Holmes (surpassing GPT-4o) with consistent gains on both reasoning and general video understanding tasks. Ablation studies verify the complementary roles of the attribution signals and the robustness of targeted token-level updates. Overall, Video-KTR improves accuracy and interpretability, offering a simple, drop-in extension to RL for complex video reasoning. Our code and models are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2601.19686
PDF
