Video_Caption
-
SAVA-X: Ego-to-Exo Imitation Error Detection via Scene-Adaptive View Alignment and Bidirectional Cross View Fusion
2026-03-13 08:09:12Xiang Li, Heqian Qiu, Lanxiao Wang, Benliu Qiu, Fanman Meng, Linfeng Xu, Hongliang LiAbstract
Error detection is crucial in industrial training, healthcare, and assembly quality control. Most existing work assumes a single-view setting and cannot handle the practical case where a third-person (exo) demonstration is used to assess a first-person (ego) imitation. We formalize Ego$\rightarrow$Exo Imitation Error Detection: given asynchronous, length-mismatched ego and exo videos, the model must localize procedural steps on the ego timeline and decide whether each is erroneous. This setting introduces cross-view domain shift, temporal misalignment, and heavy redundancy. Under a unified protocol, we adapt strong baselines from dense video captioning and temporal action detection and show that they struggle in this cross-view regime. We then propose SAVA-X, an Align-Fuse-Detect framework with (i) view-conditioned adaptive sampling, (ii) scene-adaptive view embeddings, and (iii) bidirectional cross-attention fusion. On the EgoMe benchmark, SAVA-X consistently improves AUPRC and mean tIoU over all baselines, and ablations confirm the complementary benefits of its components. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.12764
PDF
-
VCBench: A Streaming Counting Benchmark for Spatial-Temporal State Maintenance in Long Videos
2026-03-13 06:28:46Pengyiang Liu, Zhongyue Shi, Hongye Hao, Qi Fu, Xueting Bi, Siwei Zhang, Xiaoyang Hu, Zitian Wang, Linjiang Huang, Si LiuAbstract
Video understanding requires models to continuously track and update world state during playback. While existing benchmarks have advanced video understanding evaluation across multiple dimensions, the observation of how models maintain world state remains insufficient. We propose VCBench, a streaming counting benchmark that repositions counting as a minimal probe for diagnosing world state maintenance capability. We decompose this capability into object counting (tracking currently visible objects vs.\ tracking cumulative unique identities) and event counting (detecting instantaneous actions vs.\ tracking complete activity cycles), forming 8 fine-grained subcategories. VCBench contains 406 videos with frame-by-frame annotations of 10,071 event occurrence moments and object state change moments, generating 1,000 streaming QA pairs with 4,576 query points along timelines. By observing state maintenance trajectories through streaming multi-point queries, we design three complementary metrics to diagnose numerical precision, trajectory consistency, and temporal awareness. Evaluation on mainstream video-language models shows that current models still exhibit significant deficiencies in spatial-temporal state maintenance, particularly struggling with tasks like periodic event counting. VCBench provides a diagnostic framework for measuring and improving state maintenance in video understanding systems.
Abstract (translated)
URL
https://arxiv.org/abs/2603.12703
PDF
-
Less Data, Faster Convergence: Goal-Driven Data Optimization for Multimodal Instruction Tuning
2026-03-12 21:54:50Rujie Wu, Haozhe Zhao, Hai Ci, Yizhou WangAbstract
Multimodal instruction tuning is often compute-inefficient because training budgets are spread across large mixed image-video pools whose utility is highly uneven. We present Goal-Driven Data Optimization (GDO), a framework that computes six sample descriptors for each candidate and constructs optimized 1$\times$ training subsets for different goals. Under a fixed one-epoch Qwen3-VL-8B-Instruct training and evaluation recipe on 8 H20 GPUs, GDO uses far fewer training samples than the Uni-10x baseline while converging faster and achieving higher accuracy. Relative to the fixed 512k-sample Uni-10x baseline, GDO reaches the Uni-10x reference after 35.4k samples on MVBench, 26.6k on VideoMME, 27.3k on MLVU, and 34.7k on LVBench, while improving Accuracy by +1.38, +1.67, +3.08, and +0.84 percentage points, respectively. The gains are largest on MVBench and MLVU, while LVBench improves more modestly, consistent with its ultra-long-video setting and the mismatch between that benchmark and the short-video/image-dominant training pool. Across MinLoss, Diverse, Temp, and Temp+, stronger temporal emphasis yields steadily better long-video understanding behavior. Overall, GDO provides a goal-driven data optimization framework that enables faster convergence with fewer training samples under a fixed training protocol. Code is available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2603.12478
PDF
-
SPARROW: Learning Spatial Precision and Temporal Referential Consistency in Pixel-Grounded Video MLLMs
2026-03-12 18:59:57Mohamad Alansari, Naufal Suryanto, Divya Velayudhan, Sajid Javed, Naoufel Werghi, Muzammal NaseerAbstract
Multimodal large language models (MLLMs) have advanced from image-level reasoning to pixel-level grounding, but extending these capabilities to videos remains challenging as models must achieve spatial precision and temporally consistent reference tracking. Existing video MLLMs often rely on a static segmentation token ([SEG]) for frame-wise grounding, which provides semantics but lacks temporal context, causing spatial drift, identity switches, and unstable initialization when objects move or reappear. We introduce SPARROW, a pixel-grounded video MLLM that unifies spatial accuracy and temporal stability through two key components: (i) Target-Specific Tracked Features (TSF), which inject temporally aligned referent cues during training, and (ii) a dual-prompt design that decodes box ([BOX]) and segmentation ([SEG]) tokens to fuse geometric priors with semantic grounding. SPARROW is supported by a curated referential video dataset of 30,646 videos and 45,231 Q&A pairs and operates end-to-end without external detectors via a class-agnostic SAM2-based proposer. Integrated into three recent open-source video MLLMs (UniPixel, GLUS, and VideoGLaMM), SPARROW delivers consistent gains across six benchmarks, improving up to +8.9 J&F on RVOS, +5 mIoU on visual grounding, and +5.4 CLAIR on GCG. These results demonstrate that SPARROW substantially improves referential stability, spatial precision, and temporal coherence in pixel-grounded video understanding. Project page: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2603.12382
PDF
-
Video Streaming Thinking: VideoLLMs Can Watch and Think Simultaneously
2026-03-12 17:59:51Yiran Guan, Liang Yin, Dingkang Liang, Jianzhong Ju, Zhenbo Luo, Jian Luan, Yuliang Liu, Xiang BaiAbstract
Online Video Large Language Models (VideoLLMs) play a critical role in supporting responsive, real-time interaction. Existing methods focus on streaming perception, lacking a synchronized logical reasoning stream. However, directly applying test-time scaling methods incurs unacceptable response latency. To address this trade-off, we propose Video Streaming Thinking (VST), a novel paradigm for streaming video understanding. It supports a thinking while watching mechanism, which activates reasoning over incoming video clips during streaming. This design improves timely comprehension and coherent cognition while preserving real-time responsiveness by amortizing LLM reasoning latency over video playback. Furthermore, we introduce a comprehensive post-training pipeline that integrates VST-SFT, which structurally adapts the offline VideoLLM to causal streaming reasoning, and VST-RL, which provides end-to-end improvement through self-exploration in a multi-turn video interaction environment. Additionally, we devise an automated training-data synthesis pipeline that uses video knowledge graphs to generate high-quality streaming QA pairs, with an entity-relation grounded streaming Chain-of-Thought to enforce multi-evidence reasoning and sustained attention to the video stream. Extensive evaluations show that VST-7B performs strongly on online benchmarks, e.g. 79.5% on StreamingBench and 59.3% on OVO-Bench. Meanwhile, VST remains competitive on offline long-form or reasoning benchmarks. Compared with Video-R1, VST responds 15.7 times faster and achieves +5.4% improvement on VideoHolmes, demonstrating higher efficiency and strong generalization across diverse video understanding tasks. Code, data, and models will be released at this https URL.
Abstract (translated)
在线视频大型语言模型（VideoLLMs）在支持响应式、实时交互中扮演着关键角色。现有方法主要集中在流媒体感知上，缺乏同步的逻辑推理过程。然而，直接应用测试时间缩放方法会导致无法接受的响应延迟。为解决这一权衡问题，我们提出了视频流思考（VST），这是一种新的用于视频理解的流媒体范式。它支持边看边想机制，在流媒体期间激活对传入视频片段的推理。这种设计通过将大型语言模型的推理延迟分摊到视频播放中来提高及时理解和连贯的认知，并保持实时响应性。 此外，我们还引入了一个全面的后训练流水线，该流水线整合了VST-SFT和VST-RL。VST-SFT结构性地将离线VideoLLM适应为因果流推理，而VST-RL则通过在多回合视频互动环境中自我探索提供端到端改进。我们还设计了一个自动化的训练数据合成流水线，该流水线使用视频知识图来生成高质量的流媒体问答对，并采用基于实体关系的连贯思考链来强制执行多重证据推理和对视频流的持续关注。 广泛的评估表明，VST-7B在在线基准测试（如StreamingBench上的79.5%和OVO-Bench上的59.3%）中表现强劲。同时，在离线长格式或推理基准上，VST也保持了竞争力。与Video-R1相比，VST的响应速度快了15.7倍，并在VideoHolmes上实现了+5.4%的改进，展示了更高的效率和跨多样视频理解任务的强大泛化能力。 代码、数据和模型将在以下网址发布：[https URL]。
URL
https://arxiv.org/abs/2603.12262
PDF
-
Attend Before Attention: Efficient and Scalable Video Understanding via Autoregressive Gazing
2026-03-12 17:58:52Baifeng Shi, Stephanie Fu, Long Lian, Hanrong Ye, David Eigen, Aaron Reite, Boyi Li, Jan Kautz, Song Han, David M. Chan, Pavlo Molchanov, Trevor Darrell, Hongxu YinAbstract
Multi-modal large language models (MLLMs) have advanced general-purpose video understanding but struggle with long, high-resolution videos -- they process every pixel equally in their vision transformers (ViTs) or LLMs despite significant spatiotemporal redundancy. We introduce AutoGaze, a lightweight module that removes redundant patches before processed by a ViT or an MLLM. Trained with next-token prediction and reinforcement learning, AutoGaze autoregressively selects a minimal set of multi-scale patches that can reconstruct the video within a user-specified error threshold, eliminating redundancy while preserving information. Empirically, AutoGaze reduces visual tokens by 4x-100x and accelerates ViTs and MLLMs by up to 19x, enabling scaling MLLMs to 1K-frame 4K-resolution videos and achieving superior results on video benchmarks (e.g., 67.0% on VideoMME). Furthermore, we introduce HLVid: the first high-resolution, long-form video QA benchmark with 5-minute 4K-resolution videos, where an MLLM scaled with AutoGaze improves over the baseline by 10.1% and outperforms the previous best MLLM by 4.5%. Project page: this https URL.
Abstract (translated)
多模态大型语言模型（MLLMs）在通用视频理解方面取得了显著进展，但在处理长、高分辨率视频时却面临挑战——它们的视觉变换器（ViTs）或大型语言模型对每个像素进行同等处理，尽管存在显著的空间和时间冗余。为了解决这一问题，我们引入了AutoGaze模块，它可以在视频被输入到ViT或MLLM之前去除冗余的图像块。通过下一令牌预测和强化学习训练，AutoGaze能够自回归地选择一组最小化尺度的补丁集，在用户指定的误差阈值内重建视频，从而在减少冗余的同时保留信息。 从实验结果来看，AutoGaze可以将视觉标记减少4到100倍，并使ViTs和MLLMs的速度提升最多19倍，使得MLLMs能够扩展至处理包含1K帧、4K分辨率的视频。此外，在诸如VideoMME等视频基准测试中取得了优于现有方法的表现（例如在VideoMME中的得分达到了67.0%）。 我们还推出了HLVid：首个高分辨率长格式视频问答基准，它包含了5分钟时长、4K分辨率的视频数据集。在这个新的基准上，使用AutoGaze扩展后的MLLM相对于基线模型提高了10.1%，并且相比之前的最佳MLLM性能提升了4.5%。 项目页面链接：[此处插入具体网址]
URL
https://arxiv.org/abs/2603.12254
PDF
-
BehaviorVLM: Unified Finetuning-Free Behavioral Understanding with Vision-Language Reasoning
2026-03-12 17:09:20Jingyang Ke, Weihan Li, Amartya Pradhan, Jeffrey Markowitz, Anqi WuAbstract
Understanding freely moving animal behavior is central to neuroscience, where pose estimation and behavioral understanding form the foundation for linking neural activity to natural actions. Yet both tasks still depend heavily on human annotation or unstable unsupervised pipelines, limiting scalability and reproducibility. We present BehaviorVLM, a unified vision-language framework for pose estimation and behavioral understanding that requires no task-specific finetuning and minimal human labeling by guiding pretrained Vision-Language Models (VLMs) through detailed, explicit, and verifiable reasoning steps. For pose estimation, we leverage quantum-dot-grounded behavioral data and propose a multi-stage pipeline that integrates temporal, spatial, and cross-view reasoning. This design greatly reduces human annotation effort, exposes low-confidence labels through geometric checks such as reprojection error, and produces labels that can later be filtered, corrected, or used to fine-tune downstream pose models. For behavioral understanding, we propose a pipeline that integrates deep embedded clustering for over-segmented behavior discovery, VLM-based per-clip video captioning, and LLM-based reasoning to merge and semantically label behavioral segments. The behavioral pipeline can operate directly from visual information and does not require keypoints to segment behavior. Together, these components enable scalable, interpretable, and label-light analysis of multi-animal behavior.
Abstract (translated)
理解自由活动动物的行为是神经科学的核心，其中姿态估计和行为理解构成了将神经活动与自然动作联系起来的基础。然而，这两项任务仍然高度依赖于人工标注或不稳定的无监督流程，从而限制了其可扩展性和再现性。我们提出了BehaviorVLM，这是一种统一的视觉-语言框架，用于姿态估计和行为理解，无需特定任务微调，并通过详细的、明确且可验证的推理步骤引导预训练的视觉-语言模型（VLMs），大大减少了对人类标注的需求。 对于姿态估计，我们利用量子点接地的行为数据并提出了一种多阶段管道，该管道集成了时间、空间和跨视图推理。这种设计极大地减少了人工注释的工作量，并通过几何检查（如重新投影误差）来揭示低置信度标签，从而生成可后续过滤、修正或用于微调下游姿态模型的标签。 对于行为理解，我们提出了一种集成深度嵌入聚类以发现过度分割的行为、基于VLM的逐片段视频描述以及LLM推理合并和语义标记行为段的管道。这种行为分析流程可以直接从视觉信息中操作，并不需要关键点来划分行为。 这些组件共同使得多动物行为的大规模、可解释且标签轻量级的分析成为可能。
URL
https://arxiv.org/abs/2603.12176
PDF
-
WAT: Online Video Understanding Needs Watching Before Thinking
2026-03-12 13:42:32Zifan Han, Hongbo Sun, Jinglin Xu, Canhui Tang, Yulong Lei, Xuchong Zhang, Hongbin Sun, Zhongjiang He, Hao SunAbstract
Multimodal Large Language Models (MLLMs) have shown strong capabilities in image understanding, motivating recent efforts to extend them to video reasoning. However, existing Video LLMs struggle in online streaming scenarios, where long temporal context must be preserved under strict memory constraints. We propose WAT (Watching Before Thinking), a two-stage framework for online video reasoning. WAT separates processing into a query-independent watching stage and a query-triggered thinking stage. The watching stage builds a hierarchical memory system with a Short-Term Memory (STM) that buffers recent frames and a fixed-capacity Long-Term Memory (LTM) that maintains a diverse summary of historical content using a redundancy-aware eviction policy. In the thinking stage, a context-aware retrieval mechanism combines the query with the current STM context to retrieve relevant historical frames from the LTM for cross-temporal reasoning. To support training for online video tasks, we introduce WAT-85K, a dataset containing streaming-style annotations emphasizing real-time perception, backward tracing, and forecasting. Experiments show that WAT achieves state-of-the-art performance on online video benchmarks, including 77.7% accuracy on StreamingBench and 55.2% on OVO-Bench, outperforming existing open-source online Video LLMs while operating at real-time frame rates.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13412
PDF
-
Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models
2026-03-12 13:13:50Lu Wang (The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China), Zhuoran Jin (The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China), Yupu Hao (The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China), Yubo Chen (The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China), Kang Liu (The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China), Yulong Ao (Beijing Academy of Artificial Intelligence), Jun Zhao (The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China)Abstract
Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, making multi-turn interaction over continuously arriving video streams difficult. Existing streaming methods typically use an interleaved perception-generation paradigm, which prevents concurrent perception and generation and leads to early memory decay as streams grow, hurting long-range dependency modeling. We propose Think While Watching, a memory-anchored streaming video reasoning framework that preserves continuous segment-level memory during multi-turn interaction. We build a three-stage, multi-round chain-of-thought dataset and adopt a stage-matched training strategy, while enforcing strict causality through a segment-level streaming causal mask and streaming positional encoding. During inference, we introduce an efficient pipeline that overlaps watching and thinking and adaptively selects the best attention backend. Under both single-round and multi-round streaming input protocols, our method achieves strong results. Built on Qwen3-VL, it improves single-round accuracy by 2.6% on StreamingBench and by 3.79% on OVO-Bench. In the multi-round setting, it maintains performance while reducing output tokens by 56%. Code is available at: this https URL
Abstract (translated)
多模态大型语言模型（MLLMs）在离线视频理解方面表现出色，但大多数都局限于离线推理或在线推理能力较弱，这使得处理连续到来的视频流中的多次交互变得困难。现有的流式方法通常采用交错感知-生成范式，这种模式阻止了并发感知与生成，并随着数据流的增长导致早期记忆衰退，影响长程依赖建模。 为此，我们提出了“边看边思考”（Think While Watching），这是一个以记忆为中心的流式视频推理框架，在多轮交互中保留连续的片段级记忆。我们建立了一个三阶段、多回合的思想链条数据集，并采用匹配阶段的训练策略，同时通过片段级别的流因果掩码和位置编码来强制执行严格的因果关系。 在推断过程中，我们引入了一种高效的流水线方法，该方法重叠了“看”与“思考”的过程，并根据需要自适应地选择最佳注意力后端。无论是单回合还是多回合的流式输入协议下，我们的方法都能取得优异的成绩。基于Qwen3-VL构建，它在StreamingBench上的单轮精度提高了2.6%，在OVO-Bench上提高了3.79%。在多轮设置中，它可以保持性能的同时减少了输出令牌数56%。 代码可在以下网址获取：[此链接](this https URL)
URL
https://arxiv.org/abs/2603.11896
PDF
-
Follow the Saliency: Supervised Saliency for Retrieval-augmented Dense Video Captioning
2026-03-12 02:37:21Seung hee Choi, MinJu Jeon, Hyunwoo Oh, Jihwan Lee, Dong-Jin KimAbstract
Existing retrieval-augmented approaches for Dense Video Captioning (DVC) often fail to achieve accurate temporal segmentation aligned with true event boundaries, as they rely on heuristic strategies that overlook ground truth event boundaries. The proposed framework, \textbf{STaRC}, overcomes this limitation by supervising frame-level saliency through a highlight detection module. Note that the highlight detection module is trained on binary labels derived directly from DVC ground truth annotations without the need for additional annotation. We also propose to utilize the saliency scores as a unified temporal signal that drives retrieval via saliency-guided segmentation and informs caption generation through explicit Saliency Prompts injected into the decoder. By enforcing saliency-constrained segmentation, our method produces temporally coherent segments that align closely with actual event transitions, leading to more accurate retrieval and contextually grounded caption generation. We conduct comprehensive evaluations on the YouCook2 and ViTT benchmarks, where STaRC achieves state-of-the-art performance across most of the metrics. Our code is available at this https URL
Abstract (translated)
现有的密集视频字幕（Dense Video Captioning，简称 DVC）增强检索方法常常难以实现与真实事件边界准确对齐的时序分割，因为它们依赖于忽视了地面真实事件边界的启发式策略。所提出的框架**STaRC**通过帧级显著性监督模块克服了这一限制，该模块用于检测亮点。需要注意的是，这个亮点检测模块是基于从 DVC 地面真相注释中直接得出的二进制标签进行训练的，并且不需要额外标注。 我们还建议利用显著性得分作为统一的时间信号，通过显著性引导分割来驱动检索，并通过显式注入解码器中的显著性提示（Saliency Prompts）为字幕生成提供信息。通过强制执行显著性约束的分割，我们的方法能够产生与实际事件转换紧密对齐的时间一致性片段，从而实现更准确的检索和基于上下文的字幕生成。 我们在 YouCook2 和 ViTT 基准测试上进行了全面评估，STaRC 在大多数指标上取得了最先进的性能。代码可在 [此处](此链接需您提供具体URL) 获取。
URL
https://arxiv.org/abs/2603.11460
PDF
-
Stay in your Lane: Role Specific Queries with Overlap Suppression Loss for Dense Video Captioning
2026-03-12 02:04:09Seung Hyup Baek, Jimin Lee, Hyeongkeun Lee, Jae Won ChoAbstract
Dense Video Captioning (DVC) is a challenging multimodal task that involves temporally localizing multiple events within a video and describing them with natural language. While query-based frameworks enable the simultaneous, end-to-end processing of localization and captioning, their reliance on shared queries often leads to significant multi-task interference between the two tasks, as well as temporal redundancy in localization. In this paper, we propose utilizing role-specific queries that separate localization and captioning into independent components, allowing each to exclusively learn its role. We then employ contrastive alignment to enforce semantic consistency between the corresponding outputs, ensuring coherent behavior across the separated queries. Furthermore, we design a novel suppression mechanism in which mutual temporal overlaps across queries are penalized to tackle temporal redundancy, supervising the model to learn distinct, non-overlapping event regions for more precise localization. Additionally, we introduce a lightweight module that captures core event concepts to further enhance semantic richness in captions through concept-level representations. We demonstrate the effectiveness of our method through extensive experiments on major DVC benchmarks YouCook2 and ActivityNet Captions.
Abstract (translated)
密集视频字幕（Dense Video Captioning，DVC）是一项具有挑战性的多模态任务，涉及在视频中定位多个事件并用自然语言描述它们。虽然基于查询的框架能够同时、端到端地处理定位和字幕生成这两个任务，但这些方法通常依赖于共享查询，这往往会导致两个任务之间存在显著的任务干扰以及定位过程中出现的时间冗余。在这篇论文中，我们提出使用角色特定查询的方法，将定位与字幕生成分离开来作为独立的组件，使每个组件能够专注于学习其自身的功能。随后，我们采用对比对齐方法以强制执行对应输出之间的语义一致性，确保分离后的查询之间的一致行为。此外，我们设计了一种新颖的抑制机制，在此机制中跨查询的时间重叠被惩罚，用以解决时间冗余问题，并监督模型为更精准定位学习到不相交的独特事件区域。另外，我们还引入了一个轻量级模块来捕捉核心事件概念，通过概念级别的表示进一步增强字幕中的语义丰富性。 我们在主要的DVC基准测试YouCook2和ActivityNet Captions上进行了广泛的实验，并展示了我们的方法的有效性。
URL
https://arxiv.org/abs/2603.11439
PDF
-
Beyond Single-Sample: Reliable Multi-Sample Distillation for Video Understanding
2026-03-12 01:30:37Songlin Li, Xin Zhu, Zechao Guan, Peipeng Chen, Jian YaoAbstract
Traditional black-box distillation for Large Vision-Language Models (LVLMs) typically relies on a single teacher response per input, which often yields high-variance responses and format inconsistencies in multimodal or temporal scenarios. To mitigate this unreliable supervision, we propose R-MSD (Reliable Multi-Sample Distillation), a framework that explicitly models teacher sampling variance to enhance distillation stability. Rather than relying on a single teacher response, our approach leverages a task-adaptive teacher pool to provide robust supervision tailored to both closed-ended and open-ended reasoning. By integrating quality-aware signal matching with an adversarial distillation objective, our approach effectively filters teacher noise while maximizing knowledge transfer. Extensive evaluations across comprehensive video understanding benchmarks demonstrate that R-MSD consistently outperforms single sample distillation methods. We additionally include an original SFT+RL 4B baseline under the same training budget, which shows only marginal gains, while our method achieves significant improvements. With a 4B student model, our approach delivers gains on VideoMME (+1.5%), Video-MMMU (+3.2%), and MathVerse (+3.6%).
Abstract (translated)
传统黑盒蒸馏（Black-box distillation）对于大型视觉-语言模型（Large Vision-Language Models，LVLMs）通常依赖于每个输入只有一个教师响应的方法，这在多模态或多时间场景中往往会导致高方差的响应和格式不一致的问题。为了解决这种不可靠的监督问题，我们提出了R-MSD（Reliable Multi-Sample Distillation），这是一个通过显式建模教师样本变异性来增强蒸馏稳定性的框架。 与依赖单一教师响应不同，我们的方法利用了一个任务适应性教师池，以提供针对封闭和开放类型推理都能有效支持的稳健监督。通过整合质量感知信号匹配以及对抗性蒸馏目标，我们的方法能够有效地过滤掉教师噪声并最大化知识传递效率。 广泛的评估显示，在全面覆盖视频理解基准测试时，R-MSD始终优于单一样本蒸馏方法。我们还提供了一个原始SFT+RL 4B基线，该基线在相同训练预算下仅显示出边际改善，而我们的方法实现了显著的改进。使用一个40亿参数的学生模型，我们的方法分别在VideoMME（+1.5%）、Video-MMMU（+3.2%）和MathVerse（+3.6%）上取得了提升效果。 总结来说，R-MSD通过有效管理教师样本变异性，并结合质量感知信号匹配与对抗性蒸馏目标来增强知识的稳定传递，在大规模多模态任务中展现出优越的表现。
URL
https://arxiv.org/abs/2603.11423
PDF
-
TennisExpert: Towards Expert-Level Analytical Sports Video Understanding
2026-03-11 15:12:10Zhaoyu Liu, Xi Weng, Lianyu Hu, Zhe Hou, Kan Jiang, Jin Song Dong, Yang LiuAbstract
Tennis is one of the most widely followed sports, generating extensive broadcast footage with strong potential for professional analysis, automated coaching, and real-time commentary. However, automatic tennis understanding remains underexplored due to two key challenges: (1) the lack of large-scale benchmarks with fine-grained annotations and expert-level commentary, and (2) the difficulty of building accurate yet efficient multimodal systems suitable for real-time deployment. To address these challenges, we introduce TennisVL, a large-scale tennis benchmark comprising over 200 professional matches (471.9 hours) and 40,000+ rally-level clips. Unlike existing commentary datasets that focus on descriptive play-by-play narration, TennisVL emphasizes expert analytical commentary capturing tactical reasoning, player decisions, and match momentum. Furthermore, we propose TennisExpert, a multimodal tennis understanding framework that integrates a video semantic parser with a memory-augmented model built on Qwen3-VL-8B. The parser extracts key match elements (e.g., scores, shot sequences, ball bounces, and player locations), while hierarchical memory modules capture both short- and long-term temporal context. Experiments show that TennisExpert consistently outperforms strong proprietary baselines, including GPT-5, Gemini, and Claude, and demonstrates improved ability to capture tactical context and match dynamics.
Abstract (translated)
URL
https://arxiv.org/abs/2603.13397
PDF
-
SurgFed: Language-guided Multi-Task Federated Learning for Surgical Video Understanding
2026-03-10 10:54:14Zheng Fang, Ziwei Niu, Ziyue Wang, Zhu Zhuo, Haofeng Liu, Shuyang Qian, Jun Xia, Yueming JinAbstract
Surgical scene Multi-Task Federated Learning (MTFL) is essential for robot-assisted minimally invasive surgery (RAS) but remains underexplored in surgical video understanding due to two key challenges: (1) Tissue Diversity: Local models struggle to adapt to site-specific tissue features, limiting their effectiveness in heterogeneous clinical environments and leading to poor local predictions. (2) Task Diversity: Server-side aggregation, relying solely on gradient-based clustering, often produces suboptimal or incorrect parameter updates due to inter-site task heterogeneity, resulting in inaccurate localization. In light of these two issues, we propose SurgFed, a multi-task federated learning framework, enabling federated learning for surgical scene segmentation and depth estimation across diverse surgical types. SurgFed is powered by two appealing designs, i.e., Language-guided Channel Selection (LCS) and Language-guided Hyper Aggregation (LHA), to address the challenge of fully exploration on corss-site and cross-task. Technically, the LCS is first designed a lightweight personalized channel selection network that enhances site-specific adaptation using pre-defined text inputs, which optimally the local model learn the specific embeddings. We further introduce the LHA that employs a layer-wise cross-attention mechanism with pre-defined text inputs to model task interactions across sites and guide a hypernetwork for personalized parameter updates. Extensive empirical evidence shows that SurgFed yields improvements over the state-of-the-art methods in five public datasets across four surgical types. The code is available at this https URL.
Abstract (translated)
手术场景下的多任务联邦学习（MTFL）对于机器人辅助的微创外科手术（RAS）至关重要，但在手术视频理解领域中由于两个关键挑战仍处于未充分探索的状态：(1) 组织多样性：局部模型难以适应特定位置的组织特征，这在异质性临床环境中限制了其效果，并导致预测不佳。(2) 任务多样性：服务器端聚合仅依赖于基于梯度的聚类，在跨站点任务差异的情况下通常会产生次优或错误的参数更新，从而导致定位不准确。鉴于这两个问题，我们提出了SurgFed，这是一种多任务联邦学习框架，使手术场景分割和深度估计在不同类型的手术之间进行分布式联邦学习成为可能。 SurgFed有两个吸引人的设计：语言引导通道选择（LCS）和语言引导超聚合（LHA），以应对跨站点和跨任务充分探索的挑战。技术上来说，首先设计了一个轻量级个性化通道选择网络——LCS，它通过预定义的文本输入增强了特定位置的适应性，并使局部模型能够学习到最合适的嵌入表示。进一步引入了LHA，该机制使用带有预定义文本输入的逐层交叉注意机制来建模跨站点的任务交互并引导超网络进行个性化参数更新。 大量的实验证据表明，在涵盖四种手术类型中的五个公开数据集上，SurgFed相对于现有方法实现了显著改进。代码可在此链接获取：[https URL]（请将[https URL]替换为实际的网址）。
URL
https://arxiv.org/abs/2603.09496
PDF
-
Fish Audio S2 Technical Report
2026-03-09 18:34:33Shijia Liao, Yuxuan Wang, Songting Liu, Yifan Cheng, Ruoyi Zhang, Tianyu Li, Shidong Li, Yisheng Zheng, Xingwei Liu, Qingzheng Wang, Zhizhuo Zhou, Jiahua Liu, Xin Chen, Dawei HanAbstract
We introduce Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions. To scale training, we develop a multi-stage training recipe together with a staged data pipeline covering video captioning and speech captioning, voice-quality assessment, and reward modeling. To push the frontier of open-source TTS, we release our model weights, fine-tuning code, and an SGLang-based inference engine. The inference engine is production-ready for streaming, achieving an RTF of 0.195 and a time-to-first-audio below 100 this http URL code and weights are available on GitHub (this https URL) and Hugging Face (this https URL). We highly encourage readers to visit this https URL to try custom voices.
Abstract (translated)
我们推出了Fish Audio S2，这是一个开源的文本到语音系统，具有多说话人、多轮对话生成功能，并且最重要的是支持通过自然语言描述进行指令控制。为了扩大训练规模，我们开发了一套多阶段训练流程以及包括视频字幕、语音质量评估和奖励建模在内的分阶段数据管道。为了推动开源TTS的前沿发展，我们发布了我们的模型权重、微调代码及基于SGLang的推理引擎。该推理引擎已经为流媒体做好了生产准备，实现了0.195的实时因子（RTF）以及不到100毫秒的第一音频响应时间。相关代码和权重可以在GitHub (此链接) 和Hugging Face (此链接) 上获取。我们强烈建议读者访问此链接以尝试自定义语音。 请注意，原文中的“this http URL”是用于占位符的文本，实际使用时应替换为具体的URL链接地址。
URL
https://arxiv.org/abs/2603.08823
PDF
-
StreamReady: Learning What to Answer and When in Long Streaming Videos
2026-03-09 17:02:44Shehreen Azad, Vibhav Vineet, Yogesh Singh RawatAbstract
Streaming video understanding often involves time-sensitive scenarios where models need to answer exactly when the supporting visual evidence appears: answering before the evidence reflects speculation, answering after it has passed reduces real-time utility. To capture this behavior, we introduce a readiness-aware formulation of streaming video understanding with the Answer Readiness Score (ARS), a timing-aware objective with asymmetric early and late penalties. When combined with correctness, ARS defines an effective accuracy that measures not just whether a model is right, but whether it answers at the appropriate moment. Building on this formulation, we introduce StreamReady, a framework to unify temporal reasoning with on-time answering through a lightweight readiness mechanism that decides if sufficient evidence has been observed before responding. To evaluate this capability, we further introduce ProReady-QA, a benchmark with annotated answer evidence windows and proactive multi-turn questions across local and global contexts. StreamReady achieves superior performance on ProReady-QA, and consistently outperforms prior methods across eight additional streaming and offline long-video benchmarks, demonstrating robust and broadly generalizable video understanding capability.
Abstract (translated)
流媒体视频理解通常涉及时间敏感的场景，模型需要在支持性视觉证据出现时准确作答：如果提前于证据出现之前回答，则视为推测；而延迟到证据已过去才作答则降低了实时效用。为捕捉这一行为特征，我们引入了一种具有时间感知目标功能的流媒体视频理解框架——答案准备度得分（ARS），该框架包含不对称的早出和迟答惩罚机制。结合正确性考量后，ARS定义了一个有效准确率指标，不仅衡量模型是否回答正确，还评估其回答时机是否恰当。在此基础上，我们提出了StreamReady框架，它通过轻量级的答案准备度机制统一了时间推理与及时作答功能，决定何时累积足够证据后再作出响应。 为了评估这一能力，我们进一步引入了ProReady-QA基准测试集，该测试集包含标注好的答案依据窗口及跨局部和全局上下文的主动多轮问答。StreamReady在ProReady-QA上取得了卓越的表现，并且在八项额外的流媒体与离线长视频基准测试中均持续优于先前的方法，展示了其稳健而广泛适用的视频理解能力。
URL
https://arxiv.org/abs/2603.08620
PDF
-
Dynin-Omni: Omnimodal Unified Large Diffusion Language Model
2026-03-09 13:59:02Jaeik Kim, Woojin Kim, Jihwan Hong, Yejoon Lee, Sieun Hyeon, Mintaek Lim, Yunseok Han, Dogeun Kim, Hoeun Lee, Hyunggeun Kim, Jaeyoung DoAbstract
We present Dynin-Omni, the first masked-diffusion-based omnimodal foundation model that unifies text, image, and speech understanding and generation, together with video understanding, within a single architecture. Unlike autoregressive unified models that serialize heterogeneous modalities, or compositional unified models that require orchestration with external modality-specific decoders, Dynin-Omni natively formulates omnimodal modeling as masked diffusion over a shared discrete token space, enabling iterative refinement under bidirectional context. Dynin-Omni adopts a multi-stage training strategy with model-merging-based modality expansion and omnimodal alignment. We evaluate Dynin-Omni across 19 multimodal benchmarks spanning language reasoning, image generation and editing, video understanding, and speech recognition and synthesis. Dynin-Omni achieves 87.6 on GSM8K, 1733.6 on MME-P, 61.4 on VideoMME, 0.87 on GenEval, and 2.1 WER on LibriSpeech test-clean, consistently outperforming existing open-source unified models while remaining competitive with strong modality-specific expert systems. These results demonstrate the potential of masked diffusion as a unified paradigm for any-to-any modeling, providing a flexible foundation for real-time omnimodal systems, unified cross-modal retrieval and generation, and embodied multimodal agents.
Abstract (translated)
我们推出Dynin-Omni，这是首个基于掩码扩散的全模态基础模型，能够在单一架构内统一文本、图像和语音的理解与生成，同时涵盖视频理解。与将异质模态序列化的自回归统一模型，或需要外部模态特定解码器编排的组合式统一模型不同，Dynin-Omni将全模态建模原生地构建为共享离散词元空间上的掩码扩散，支持在双向上下文中的迭代精炼。Dynin-Omni采用基于模型合并的模态扩展与全模态对齐的多阶段训练策略。我们在涵盖语言推理、图像生成与编辑、视频理解、语音识别与合成的19个多模态基准上评估Dynin-Omni。Dynin-Omni在GSM8K上达到87.6，在MME-P上达到1733.6，在VideoMME上达到61.4，在GenEval上达到0.87，在LibriSpeech test-clean上达到2.1 WER，持续优于现有开源统一模型，同时与强大的模态特定专家系统保持竞争力。这些结果证明了掩码扩散作为任意到任意建模统一范式的潜力，为实时全模态系统、统一跨模态检索与生成以及具身多模态智能体提供了灵活的基础。
URL
https://arxiv.org/abs/2604.00007
PDF
-
VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
2026-03-07 07:06:34Xueqing Yu, Bohan Li, Yan Li, Zhenheng YangAbstract
Recent Vision-Language Models (VLMs) have made remarkable progress in multimodal understanding tasks, yet their evaluation on long video understanding remains unreliable. Due to limited frame inputs, key frames necessary for answering the question may be missing from the model's input. However, models that truthfully refuse to answer under such uncertainty are marked as incorrect, while those that guess may coincidentally produce the correct answer and thus obtain deceptively higher accuracy, leading to misleading evaluation results and encouraging models to guess rather than respond honestly. To address this issue, we introduce VirtueBench, a benchmark explicitly designed to assess model trustworthiness under uncertainty. VirtueBench constructs multiple frame-sampling levels for each video and provides ground truths that distinguish between answerable and unanswerable cases. Evaluations on 25 open-source and commercial VLMs reveal distinct refusal behaviors across different model families, with refusal accuracy ranging from over 70% in the best models to nearly 0% in the worst. Moreover, most models exhibit a substantial drop in refusal when the prompt does not explicitly require them to do so. These findings highlight the need for developing trustworthy VLMs for multimodal understanding, guided by benchmarks and leaderboards that emphasize reliability and trustworthiness.
Abstract (translated)
最近的视觉-语言模型（VLM）在多模态理解任务中取得了显著进展，但在处理长视频理解时其评估仍不可靠。由于输入帧数量有限，关键帧可能未能被纳入模型的输入之中，而这些关键帧正是回答问题所需的。然而，在这种不确定性下，那些诚实地拒绝作答的模型会被认为是错误的，相反地，那些猜测答案的模型可能会偶然给出正确答案，并因此获得不实的高准确率评价，导致误导性的评估结果并鼓励模型去猜而不是诚实回应。为了解决这一问题，我们引入了VirtueBench，这是一个专门用于在不确定性环境下评估模型可信度的基准测试。 VirtueBench 为每个视频构建多个帧采样层级，并提供明确区分可回答和不可回答情况的事实依据。对25种开源及商业视觉-语言模型进行评测的结果显示，不同模型家族的表现差异显著，拒绝准确率从最佳模型超过70%到最差模型接近于零不等。此外，在提示未明示要求它们必须拒绝作答的情况下，大多数模型的拒绝率都会大幅下降。 这些发现强调了开发可靠的视觉-语言模型对于多模态理解的重要性，并指出了需要通过注重可靠性和可信度的基准测试和排行榜来引导这一发展方向。
URL
https://arxiv.org/abs/2603.07071
PDF
-
Penguin-VL: Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
2026-03-06 18:58:04Boqiang Zhang, Lei Ke, Ruihan Yang, Qi Gao, Tianyuan Qu, Rossell Chen, Dong Yu, LeoweiliangAbstract
Vision Language Model (VLM) development has largely relied on scaling model size, which hinders deployment on compute-constrained mobile and edge devices such as smartphones and robots. In this work, we explore the performance limits of compact (e.g., 2B and 8B) VLMs. We challenge the prevailing practice that state-of-the-art VLMs must rely on vision encoders initialized via massive contrastive pretraining (e.g., CLIP/SigLIP). We identify an objective mismatch: contrastive learning, optimized for discrimination, enforces coarse and category-level invariances that suppress fine-grained visual cues needed for dense captioning and complex VLM reasoning. To address this issue, we present Penguin-VL, whose vision encoder is initialized from a text-only LLM. Our experiments reveal that Penguin-Encoder serves as a superior alternative to traditional contrastive pretraining, unlocking a higher degree of visual fidelity and data efficiency for multimodal understanding. Across various image and video benchmarks, Penguin-VL achieves performance comparable to leading VLMs (e.g., Qwen3-VL) in mathematical reasoning and surpasses them in tasks such as document understanding, visual knowledge, and multi-perspective video understanding. Notably, these gains are achieved with a lightweight architecture, demonstrating that improved visual representation rather than model scaling is the primary driver of performance. Our ablations show that Penguin-Encoder consistently outperforms contrastive-pretrained encoders, preserving fine-grained spatial and temporal cues that are critical for dense perception and complex reasoning. This makes it a strong drop-in alternative for compute-efficient VLMs and enables high performance in resource-constrained settings. Code: this https URL
Abstract (translated)
视觉语言模型（VLM）的发展很大程度上依赖于扩大模型规模，这阻碍了其在计算资源受限的移动和边缘设备（如智能手机和机器人）上的部署。在这项工作中，我们探讨了紧凑型（例如20亿参数和80亿参数）VLMs性能极限的问题。我们挑战了一种流行做法：最先进的VLM必须依赖于通过大规模对比预训练（如CLIP/SigLIP）初始化的视觉编码器。我们发现了一个目标上的不匹配点：对比学习，专注于区分性优化，强加了粗粒度和类别级别的不变性抑制，阻碍了密集描述和复杂VLM推理所需的细粒度视觉线索。 为了解决这个问题，我们介绍了Penguin-VL，其视觉编码器是从文本语言模型初始化的。我们的实验表明，Penguin-Encoder作为传统对比预训练的一个更优替代方案，解锁了多模态理解中更高的视觉真实性和数据效率。在各种图像和视频基准上，Penguin-VL达到了与最先进的VLM（例如Qwen3-VL）相当的数学推理性能，并且在文档理解、视觉知识以及多视角视频理解等任务上超过了它们的表现。值得注意的是，这些改进是通过一个轻量级架构实现的，这表明更好的视觉表示而不是模型规模扩展才是提高性能的主要驱动因素。 我们的消融研究显示，Penguin-Encoder始终优于对比预训练编码器，在密集感知和复杂推理中保持了关键的细粒度空间和时间线索。这使其成为计算效率型VLM的一个强有力的替代选择，并在资源受限环境中也能实现高性能。代码：[请在这里插入实际链接]
URL
https://arxiv.org/abs/2603.06569
PDF
-
EgoReasoner: Learning Egocentric 4D Reasoning via Task-Adaptive Structured Thinking
2026-03-06 18:49:04Fangrui Zhu, Yunfeng Xi, Jianmo Ni, Mu Cai, Boqing Gong, Long Zhao, Chen Qu, Ian Miao, Yi Li, Cheng Zhong, Huaizu Jiang, Shwetak PatelAbstract
Egocentric video understanding is inherently complex due to the dynamic 4D nature of the environment, where camera motion and object displacements necessitate a continuous re-evaluation of spatial relations. In this work, we target a suite of under-explored egocentric 4D reasoning tasks, including fixture interaction counting, viewpoint-relative fixture location, object movement itinerary tracking, and stationary object localization, that require fundamentally different cognitive operations: spatial anchoring, temporal tracking, and duration reasoning. We observe that these structural differences make task-agnostic approaches insufficient: generic Chain-of-Thought methods lack task-appropriate reasoning primitives, and uniform reinforcement learning actively destabilizes performance on spatial tasks. To address this, we propose EgoReasoner, a two-stage framework that aligns both the reasoning scaffold and the reward signal to each task's cognitive structure. In the first stage, Task-Adaptive Thinking Templates guide the synthesis of structured CoT traces that teach the model to reason adaptively across task types via supervised fine-tuning. In the second stage, task-aware reward functions verify entity grounding, temporal alignment, and task-adaptive logical consistency, selectively strengthening each reasoning pathway via reinforcement fine-tuning with GRPO. Our 3B-parameter model, trained on only 16K samples, achieves 37.5% average accuracy on the challenging HD-EPIC benchmark, surpassing Qwen2.5-VL-7B (25.7%) by over 10 points.
Abstract (translated)
自拍视频理解由于环境的动态四维特性而变得复杂，相机移动和物体位移需要持续重新评估空间关系。在这项工作中，我们针对一系列未充分研究的自拍4D推理任务，包括固定装置交互计数、视角相对固定装置位置、对象运动轨迹跟踪以及静态对象定位。这些任务需要不同的认知操作：空间锚定、时间追踪及持续时间推理。我们观察到这些结构上的差异使得通用方法变得不足：一般的链式思维方法缺乏适合特定任务的推理原语，统一强化学习反而在空间任务上主动破坏了性能稳定性。为了应对这一挑战，我们提出了EgoReasoner，这是一个两阶段框架，它将推理架构和奖励信号与每个任务的认知结构对齐。第一阶段中，任务自适应思考模板指导生成结构化的CoT（链式思维）痕迹，通过监督微调让模型能够根据不同类型的任务灵活地进行推理。第二阶段中，基于任务的奖励函数验证实体接地、时间一致性及逻辑一致性，并通过GRPO强化学习选择性加强每个推理路径。 我们的30亿参数模型，在仅使用16K样本训练后，能够在具有挑战性的HD-EPIC基准测试上达到平均准确率为37.5%，超过了Qwen2.5-VL-7B（准确率25.7%）约10个百分点。
URL
https://arxiv.org/abs/2603.06561
PDF
