Video_Caption
-
MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
2026-03-05 21:43:02Dannong Xu, Zhongyu Yang, Jun Chen, Yingfang Yuan, Ming Hu, Lei Sun, Luc Van Gool, Danda Pani Paudel, Chun-Mei FengAbstract
Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.
Abstract (translated)
多模态大型语言模型（MLLMs）在评估文本、图像或视频理解的基准测试中表现出色。然而，这些设置未能评价一个关键的实际需求：即从大规模、异质化的多模态语料库中检索相关证据前进行推理的需求。现有的大多数基准测试限制了检索范围为小型单一模态候选集，这大大简化了搜索空间，并夸大了端到端的可靠性。为了填补这一空白，我们引入了MultiHaystack，这是首个旨在评估大规模跨模态条件下的检索和推理能力的基准。 MultiHaystack包含超过46,000个跨文档、图像和视频的多模态检索候选对象，以及747个开放且可验证的问题。每个问题都基于检索池中的一个独特的已验证证据项，需要跨模式定位证据，并进行细致推理。在我们的研究中，我们发现当模型被提供相应证据时表现良好，但它们从整个语料库中检索这些证据的能力下降幅度很大。此外，即使最强的检索器E5-V也仅实现了40.8%的召回率@1（Recall@1），而像GPT-5这样的最先进的MLLM在获得对应证据的情况下推理准确率为80.86%，但在最接近的五个检索结果中这个数值下降到了51.4%。这些结果显示，多模态异构池中的检索仍然是MLLM的主要瓶颈，并且将MultiHaystack定位为一个有价值的测试平台，揭示了由小规模评估所掩盖的基础限制，并促进了以检索为中心的多模态系统的发展。
URL
https://arxiv.org/abs/2603.05697
PDF
-
Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline
2026-03-05 18:52:12Guo Chen, Lidong Lu, Yicheng Liu, Liangrui Dong, Lidong Zou, Jixin Lv, Zhenquan Li, Xinyi Mao, Baoqi Pei, Shihao Wang, Zhiqi Li, Karan Sapra, Fuxiao Liu, Yin-Dong Zheng, Yifei Huang, Limin Wang, Zhiding Yu, Andrew Tao, Guilin Liu, Tong LuAbstract
While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.
Abstract (translated)
尽管视频理解的数据集已经扩展到长达数小时的时长，但这些数据集通常由密集连接的片段组成，这与自然、无剧本的日常生活存在差异。为了弥合这一差距，我们引入了MM-Lifelong数据集，该数据集旨在用于多模态终身理解。它包含181.1小时的视频资料，并按天、周和月的时间尺度结构化，以捕捉不同时间密度的变化。全面评估揭示了当前范式中的两个关键失败模式：端到端MLLM（多模态终身学习模型）因上下文饱和而遭受工作记忆瓶颈问题；代表性代理基线在处理稀疏的长达一个月的时间线上则遭遇全局定位崩溃问题。 为了解决这些问题，我们提出了一种递归多模态智能体(Recursive Multimodal Agent, ReMA)，该智能体采用动态内存管理技术来迭代更新一个递归信念状态，从而显著优于现有的方法。最后，我们制定了数据集分割方案以隔离时间和领域偏见，为监督学习和分布外泛化研究提供了一个严格的理论基础。
URL
https://arxiv.org/abs/2603.05484
PDF
-
SAIL: Similarity-Aware Guidance and Inter-Caption Augmentation-based Learning for Weakly-Supervised Dense Video Captioning
2026-03-05 17:59:58Ye-Chan Kim, SeungJu Cha, Si-Woo Kim, Minju Jeon, Hyungee Kim, Dong-Jin KimAbstract
Weakly-Supervised Dense Video Captioning aims to localize and describe events in videos trained only on caption annotations, without temporal boundaries. Prior work introduced an implicit supervision paradigm based on Gaussian masking and complementary captioning. However, existing method focuses merely on generating non-overlapping masks without considering their semantic relationship to corresponding events, resulting in simplistic, uniformly distributed masks that fail to capture semantically meaningful regions. Moreover, relying solely on ground-truth captions leads to sub-optimal performance due to the inherent sparsity of existing datasets. In this work, we propose SAIL, which constructs semantically-aware masks through cross-modal alignment. Our similarity aware training objective guides masks to emphasize video regions with high similarity to their corresponding event captions. Furthermore, to guide more accurate mask generation under sparse annotation settings, we introduce an LLM-based augmentation strategy that generates synthetic captions to provide additional alignment signals. These synthetic captions are incorporated through an inter-mask mechanism, providing auxiliary guidance for precise temporal localization without degrading the main objective. Experiments on ActivityNet Captions and YouCook2 demonstrate state-of-the-art performance on both captioning and localization metrics.
Abstract (translated)
弱监督密集视频字幕任务的目标是在仅使用字幕注释（而没有时间边界信息）训练的情况下，定位和描述视频中的事件。先前的工作引入了一种基于高斯掩码和互补性字幕的隐式监督范例。然而，现有的方法仅仅关注生成不重叠的掩码，并未考虑这些掩码与相应事件之间的语义关系，导致生成了简单且均匀分布的掩码，无法捕捉到具有实际意义的语义区域。此外，单纯依赖于真实标签字幕会导致在现有数据集固有的稀疏性下表现不佳。 为了解决这些问题，我们提出了SAIL（Semantic-Aware Implicit Learning）方法，该方法通过跨模态对齐来构建语义感知掩码。我们的相似度感知训练目标引导掩码突出显示视频中与相应事件字幕具有高相似性的区域。此外，为了在标注稀疏的情况下更准确地生成掩码，我们引入了一种基于大规模语言模型（LLM）的增强策略，该策略生成合成字幕以提供额外的对齐信号。这些合成字幕通过一种跨掩码机制被整合进来，在不损害主要目标的前提下提供了精确时间定位的辅助指导。 在ActivityNet Captions和YouCook2数据集上的实验表明，SAIL方法在字幕描述和事件定位两个关键指标上均达到了当前最佳性能。
URL
https://arxiv.org/abs/2603.05437
PDF
-
Think, Then Verify: A Hypothesis-Verification Multi-Agent Framework for Long Video Understanding
2026-03-05 09:16:07Zheng Wang, Haoran Chen, Haoxuan Qin, Zhipeng Wei, Tianwen Qian, Cong BaiAbstract
Long video understanding is challenging due to dense visual redundancy, long-range temporal dependencies, and the tendency of chain-of-thought and retrieval-based agents to accumulate semantic drift and correlation-driven errors. We argue that long-video reasoning should begin not with reactive retrieval, but with deliberate task formulation: the model must first articulate what must be true in the video for each candidate answer to hold. This thinking-before-finding principle motivates VideoHV-Agent, a framework that reformulates video question answering as a structured hypothesis-verification process. Based on video summaries, a Thinker rewrites answer candidates into testable hypotheses, a Judge derives a discriminative clue specifying what evidence must be checked, a Verifier grounds and tests the clue using localized, fine-grained video content, and an Answer agent integrates validated evidence to produce the final answer. Experiments on three long-video understanding benchmarks show that VideoHV-Agent achieves state-of-the-art accuracy while providing enhanced interpretability, improved logical soundness, and lower computational cost. We make our code publicly available at: this https URL.
Abstract (translated)
长视频理解具有挑战性，原因在于其密集的视觉冗余、长时间的时序依赖关系以及基于检索和链式思考代理易于积累语义漂移和相关驱动错误的趋势。我们主张，针对长视频进行推理不应从反应式的检索开始，而应首先明确任务定义：模型必须先描述候选答案在视频中成立所必需满足的情况。这一“思考前寻找”的原则激发了VideoHV-Agent框架的诞生，该框架将视频问答问题重新构建为结构化的假设验证过程。基于视频摘要，“思考者”会将答案候选人重写成可检验的假设，“法官”则推导出一种区分性线索以明确必须检查哪些证据，“验证者”通过使用局部化、细致的视频内容来落实和测试该线索，而“回答者”则整合经过验证的证据以生成最终的答案。在三个长视频理解基准上的实验表明，VideoHV-Agent实现了最先进的准确率，并提供了增强的可解释性、改进了逻辑严谨性和降低了计算成本。我们的代码已公开发布在：[此链接](this https URL)。
URL
https://arxiv.org/abs/2603.04977
PDF
-
Are Multimodal LLMs Ready for Surveillance? A Reality Check on Zero-Shot Anomaly Detection in the Wild
2026-03-05 02:00:53Shanle Yao, Armin Danesh Pazho, Narges Rashvand, Hamed TabkhiAbstract
Multimodal large language models (MLLMs) have demonstrated impressive general competence in video understanding, yet their reliability for real-world Video Anomaly Detection (VAD) remains largely unexplored. Unlike conventional pipelines relying on reconstruction or pose-based cues, MLLMs enable a paradigm shift: treating anomaly detection as a language-guided reasoning task. In this work, we systematically evaluate state-of-the-art MLLMs on the ShanghaiTech and CHAD benchmarks by reformulating VAD as a binary classification task under weak temporal supervision. We investigate how prompt specificity and temporal window lengths (1s--3s) influence performance, focusing on the precision--recall trade-off. Our findings reveal a pronounced conservative bias in zero-shot settings; while models exhibit high confidence, they disproportionately favor the 'normal' class, resulting in high precision but a recall collapse that limits practical utility. We demonstrate that class-specific instructions can significantly shift this decision boundary, improving the peak F1-score on ShanghaiTech from 0.09 to 0.64, yet recall remains a critical bottleneck. These results highlight a significant performance gap for MLLMs in noisy environments and provide a foundation for future work in recall-oriented prompting and model calibration for open-world surveillance, which demands complex video understanding and reasoning.
Abstract (translated)
多模态大型语言模型（MLLMs）在视频理解方面展示了令人印象深刻的通用能力，但它们在现实世界的视频异常检测（VAD）中的可靠性仍然鲜有研究。与传统的依赖于重建或基于姿势的线索的方法不同，MLLMs实现了范式转变：将异常检测视为一种由语言引导的推理任务。在这项工作中，我们通过重新定义VAD为一个带有弱时间监督的二分类任务，在ShanghaiTech和CHAD基准上系统地评估了最先进的MLLMs。我们探讨了提示具体性和不同时间窗口长度（1秒至3秒）对性能的影响，并重点关注精度与召回率之间的权衡。我们的发现揭示了一个明显的保守倾向：在零样本设置下，尽管模型表现出高度自信，但它们倾向于过多地偏向“正常”类别，从而导致高精度但低召回率的问题，限制了其实际应用价值。我们展示了针对特定类别的指令可以显著改变这种决策边界，在ShanghaiTech上的峰值F1分数从0.09提升到了0.64，尽管如此，召回仍然是一个关键瓶颈。这些结果突显了MLLMs在嘈杂环境中的性能差距，并为未来的研究提供了基础：通过侧重于提高召回率的提示设计和模型校准，在开放世界的监控领域中实现复杂视频理解和推理的需求。
URL
https://arxiv.org/abs/2603.04727
PDF
-
FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answering
2026-03-04 18:14:00Tatiana Zemskova, Solomon Andryushenko, Ilya Obrubov, Viktoriia Khoruzhaia, Ekaterina Eroshenko, Ekaterina Derevyanka, Dmitry YudinAbstract
The ability to understand long videos is vital for embodied intelligent agents, because their effectiveness depends on how well they can accumulate, organize, and leverage long-horizon perceptual memories. Recently, multimodal LLMs have been gaining popularity for solving the long video understanding task due to their general ability to understand natural language and to leverage world knowledge. However, as the number of frames provided to an MLLM increases, the quality of its responses tends to degrade, and inference time grows. Therefore, when using MLLMs for long video understanding, a crucial step is selecting key frames from the video to answer user queries. In this work, we develop FocusGraph, a framework for keyframe selection for question answering over long egocentric videos. It leverages a lightweight trainable Scene-Caption LLM Selector that selects query-relevant clips based on their graph-based captions, and a training-free method for selecting keyframes from these clips. Unlike existing methods, the proposed Scene-Caption LLM Selector does not rely on the original sequence of low-resolution frames; instead, it operates on a compact textual representation of the scene. We then design a training-free Patch-wise Sparse-Flow Retention (PSFR) method to select keyframes from the resulting sequence of clips, which are fed into an MLLM to produce the final answer. Together, these components enable FocusGraph to achieve state-of-the-art results on challenging egocentric long-video question answering benchmarks, including FindingDory and HourVideo, while significantly reducing inference time relative to baseline approaches.
Abstract (translated)
理解长视频对于具身智能代理来说至关重要，因为它们的效率取决于积累、组织和利用长期感知记忆的能力。最近，多模态大语言模型（MLLM）由于其理解和利用世界知识以及自然语言的一般能力，在解决长视频理解任务方面越来越受欢迎。然而，随着提供给MLLM的帧数增加，其响应质量往往会下降，并且推理时间也会增长。因此，在使用MLLM进行长视频理解时，关键步骤之一是从视频中选择与用户查询相关的关键帧。 在这项工作中，我们开发了FocusGraph，这是一个用于在长时间第一人称视角视频问答中选择关键帧的框架。它利用了一个轻量级可训练的场景描述大语言模型选择器（Scene-Caption LLM Selector），该选择器基于基于图的字幕选择与查询相关的片段，并且还设计了一种无需训练的方法从这些片段中选择关键帧。与现有方法不同，所提出的场景描述大语言模型选择器不依赖于原始低分辨率帧序列；相反，它在紧凑的文本形式的场景表示上运行。然后，我们设计了一种无训练的分块稀疏流保留（Patch-wise Sparse-Flow Retention, PSFR）方法来从生成的片段序列中选择关键帧，并将这些关键帧输入到一个MLLM中以产生最终答案。 通过结合这些组件，FocusGraph能够在具有挑战性的第一人称视角长视频问答基准测试（如FindingDory和HourVideo）上实现最先进的结果，同时相对于基线方法显著减少推理时间。
URL
https://arxiv.org/abs/2603.04349
PDF
-
RIVER: A Real-Time Interaction Benchmark for Video LLMs
2026-03-04 12:29:34Yansong Shi, Qingsong Zhao, Tianxiang Jiang, Xiangyu Zeng, Yi Wang, Limin WangAbstract
The rapid advancement of multimodal large language models has demonstrated impressive capabilities, yet nearly all operate in an offline paradigm, hindering real-time interactivity. Addressing this gap, we introduce the Real-tIme Video intERaction Bench (RIVER Bench), designed for evaluating online video comprehension. RIVER Bench introduces a novel framework comprising Retrospective Memory, Live-Perception, and Proactive Anticipation tasks, closely mimicking interactive dialogues rather than responding to entire videos at once. We conducted detailed annotations using videos from diverse sources and varying lengths, and precisely defined the real-time interactive format. Evaluations across various model categories reveal that while offline models perform well in single question-answering tasks, they struggle with real-time processing. Addressing the limitations of existing models in online video interaction, especially their deficiencies in long-term memory and future perception, we proposed a general improvement method that enables models to interact with users more flexibly in real time. We believe this work will significantly advance the development of real-time interactive video understanding models and inspire future research in this emerging field. Datasets and code are publicly available at this https URL.
Abstract (translated)
多模态大型语言模型的快速进步已经展示了其令人印象深刻的性能，然而几乎所有这些模型都运行在离线模式中，这阻碍了其实时互动能力。为解决这一问题，我们引入了实时视频互动基准测试（RIVER Bench），旨在评估在线视频理解能力。RIVER Bench 引入了一个新颖的框架，包括回顾性记忆、现场感知和前瞻性预测任务，这些任务更加接近于模拟交互对话而不是一次性回应整个视频内容。我们使用来自各种来源且长度不同的视频进行了详细的标注，并精确定义了实时互动格式。 在不同模型类别的评估中发现，尽管离线模型在单一问答任务上表现良好，但在实时处理方面却遇到了困难。针对现有模型在线视频互动中的局限性，特别是其长期记忆和未来感知的不足，我们提出了一种通用改进方法，使模型能够更灵活地进行实时互动。我们认为这项工作将显著推动实时交互式视频理解模型的发展，并激发这一新兴领域的未来研究。 数据集和代码已公开发布在以下链接：[此 URL](https://this-URL)（请替换为实际的网址）。
URL
https://arxiv.org/abs/2603.03985
PDF
-
Proact-VL: A Proactive VideoLLM for Real-Time AI Companions
2026-03-03 19:02:46Weicai Yan, Yuhong Dai, Qi Ran, Haodong Li, Wang Lin, Hao Liao, Xing Xie, Tao Jin, Jianxun LianAbstract
Proactive and real-time interactive experiences are essential for human-like AI companions, yet face three key challenges: (1) achieving low-latency inference under continuous streaming inputs, (2) autonomously deciding when to respond, and (3) controlling both quality and quantity of generated content to meet real-time constraints. In this work, we instantiate AI companions through two gaming scenarios, commentator and guide, selected for their suitability for automatic evaluation. We introduce the Live Gaming Benchmark, a large-scale dataset with three representative scenarios: solo commentary, co-commentary, and user guidance, and present Proact-VL, a general framework that shapes multimodal language models into proactive, real-time interactive agents capable of human-like environment perception and interaction. Extensive experiments show Proact-VL achieves superior response latency and quality while maintaining strong video understanding capabilities, demonstrating its practicality for real-time interactive applications.
Abstract (translated)
主动且实时的互动体验对于类人AI伴侣来说至关重要，但面临着三大挑战：(1) 在持续流式输入的情况下实现低延迟推理；(2) 自主决定何时响应；以及 (3) 控制生成内容的质量和数量以满足实时约束。在这项工作中，我们通过两个游戏场景——解说员和导游来实例化AI伴侣，并选择了这两个场景因其适合自动评估。我们引入了“Live Gaming Benchmark”，这是一个大规模的数据集，包含三个代表性场景：个人解说、联合解说以及用户指导。此外，我们提出了Proact-VL框架，该框架能够将多模态语言模型塑造成具备类人环境感知和互动能力的主动型实时交互代理。广泛的实验表明，Proact-VL在保持强大的视频理解能力的同时，实现了优越的响应延迟和质量，证明了其在实时交互应用中的实用性。
URL
https://arxiv.org/abs/2603.03447
PDF
-
Think-as-You-See: Streaming Chain-of-Thought Reasoning for Large Vision-Language Models
2026-03-03 11:24:55Jialiang Zhang, Junlong Tong, Junyan Lin, Hao Wu, Yirong Sun, Yunpu Ma, Xiaoyu ShenAbstract
Large Vision Language Models (LVLMs) exhibit strong Chain-of-Thought (CoT) capabilities, yet most existing paradigms assume full-video availability before inference, a batch-style process misaligned with real-world video streams where information arrives sequentially. Motivated by the streaming nature of video data, we investigate two streaming reasoning paradigms for LVLMs. The first, an interleaved paradigm, alternates between receiving frames and producing partial reasoning but remains constrained by strictly ordered cache updates. To better match streaming inputs, we propose \textbf{Think-as-You-See (TaYS)}, a unified framework enabling true concurrent reasoning. TaYS integrates parallelized CoT generation, stream-constrained training, and stream-parallel inference. It further employs temporally aligned reasoning units, streaming attention masks and positional encodings, and a dual KV-cache that decouples visual encoding from textual reasoning. We evaluate all paradigms on the Qwen2.5-VL family across representative video CoT tasks, including event dynamics analysis, causal reasoning, and thematic understanding. Experiments show that TaYS consistently outperforms both batch and interleaved baselines, improving reasoning performance while substantially reducing time-to-first-token (TTFT) and overall reasoning delay. These results demonstrate the effectiveness of data-aligned streaming reasoning in enabling efficient and responsive video understanding for LVLMs. We release our code at \href{this https URL}{this repository.}
Abstract (translated)
大型视觉语言模型（LVLM）表现出强大的链式思维（CoT）能力，但大多数现有方法假设在推理之前视频是完全可用的，这与实际视频流中信息依次到达的情况不一致。鉴于视频数据的数据流特性，我们研究了两种适用于LVLM的流式推理范式。第一个是交错型范式，在接收帧和生成部分推理之间交替进行，但仍受到严格顺序缓存更新的限制。为了更好地匹配流式输入，我们提出了一种名为“所见即思考（TaYS）”的统一框架，该框架实现了真正的并发推理能力。TaYS集成了并行化的CoT生成、受流限训练和流平行推断，并进一步采用了时间对齐的推理单元、流式的注意力掩码与位置编码以及双KV缓存机制，后者将视觉编码从文本推理中解耦出来。 我们在Qwen2.5-VL家族上评估了所有范式在代表性视频CoT任务上的表现，包括事件动态分析、因果推理和主题理解。实验结果表明，TaYS始终优于批量处理和交错型基准模型，在提高推理性能的同时大幅减少了首次生成令牌所需时间（TTFT）及整体推理延迟。这些结果显示，与数据一致的流式推理方法对于LVLM实现高效的视频理解和响应性具有有效性。我们将在该网址发布我们的代码：\[此链接\]
URL
https://arxiv.org/abs/2603.02872
PDF
-
FluxMem: Adaptive Hierarchical Memory for Streaming Video Understanding
2026-03-02 17:16:47Yiweng Xie, Bo He, Junke Wang, Xiangyu Zheng, Ziyi Ye, Zuxuan WuAbstract
This paper presents FluxMem, a training-free framework for efficient streaming video understanding. FluxMem adaptively compresses redundant visual memory through a hierarchical, two-stage design: (1) a Temporal Adjacency Selection (TAS) module removes redundant visual tokens across adjacent frames, and (2) a Spatial Domain Consolidation (SDC) module further merges spatially repetitive regions within each frame into compact representations. To adapt effectively to dynamic scenes, we introduce a self-adaptive token compression mechanism in both TAS and SDC, which automatically determines the compression rate based on intrinsic scene statistics rather than manual tuning. Extensive experiments demonstrate that FluxMem achieves new state-of-the-art results on existing online video benchmarks, reaching 76.4 on StreamingBench and 67.2 on OVO-Bench under real-time settings, while reducing latency by 69.9% and peak GPU memory by 34.5% on OVO-Bench. Furthermore, it maintains strong offline performance, achieving 73.1 on MLVU while using 65% fewer visual tokens.
Abstract (translated)
本文介绍了FluxMem，这是一种无需训练的高效流媒体视频理解框架。FluxMem通过一个分层、两阶段的设计自适应地压缩冗余视觉记忆：（1）时序邻近选择(TAS)模块删除相邻帧之间的冗余视觉标记；（2）空间领域合并(SDC)模块进一步将每个帧内的重复空间区域合并为紧凑的表示。为了有效适应动态场景，我们在TAS和SDC中引入了一种自适应令牌压缩机制，该机制可以根据内在场景统计自动确定压缩率，而不是手动调整。 广泛的实验表明，在现有的在线视频基准测试上，FluxMem达到了新的最佳性能：在实时设置下，在StreamingBench上的得分为76.4分，在OVO-Bench上的得分为67.2分，并且在OVO-Bench中延迟减少了69.9%，峰值GPU内存使用量减少了34.5%。此外，它还保持了强大的离线性能，在MLVU上的得分达到了73.1，同时使用的视觉令牌数量减少了65%。
URL
https://arxiv.org/abs/2603.02096
PDF
-
From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents
2026-03-02 05:12:45Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, Min Zhang, Shu-Tao XiaAbstract
While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy, which first tries with the abstract Symbolic Schema and progressively "drills down" to the Sensory Buffer and Episodic Stream under high uncertainty. Extensive experiments across 4 benchmarks confirm the effectiveness of MM-Mem on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code is available at this https URL.
Abstract (translated)
尽管多模态大型语言模型在短期推理方面表现出色，但由于有限的上下文窗口和静态记忆机制，它们在理解长时段视频时遇到困难，这些机制无法模仿人类的认知效率。现有方法通常走向两个极端：一种是以视觉为中心的方法，通过密集的视觉积累导致高延迟和冗余；另一种是基于文本的方法，由于过分强调字幕而造成细节丢失和幻觉。 为了弥合这一差距，我们提出了MM-Mem（多模态金字塔记忆架构），该架构建立在模糊痕迹理论的基础上。MM-Mem将内存结构化为三个层级：感觉缓冲区、事件流以及符号模式，从而能够逐步提炼出细微的感知痕迹（verbatim）并将其转化为高层次的语义模型（gist）。为了动态构建内存，我们推导出了一个语义信息瓶颈目标，并引入了SIB-GRPO来优化内存压缩和任务相关信息保留之间的权衡。在推理过程中，我们设计了一种基于熵驱动的自顶向下记忆检索策略，在高度不确定性情况下，首先尝试从抽象符号模式开始，然后逐步“深入”到感觉缓冲区和事件流。 通过四个基准测试的广泛实验表明了MM-Mem在离线任务和流式任务上的有效性，展示了强大的泛化能力和认知启发的记忆组织的有效性。相关代码可在该网址上获取（请将"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2603.01455
PDF
-
Token Reduction via Local and Global Contexts Optimization for Efficient Video Large Language Models
2026-03-02 03:06:40Jinlong Li, Liyuan Jiang, Haonan Zhang, Nicu SebeAbstract
Video Large Language Models (VLLMs) demonstrate strong video understanding but suffer from inefficiency due to redundant visual tokens. Existing pruning primary targets intra-frame spatial redundancy or prunes inside the LLM with shallow-layer overhead, yielding suboptimal spatiotemporal reduction and underutilizing long-context compressibility. All of them often discard subtle yet informative context from merged or pruned tokens. In this paper, we propose a new perspective that elaborates token \textbf{A}nchors within intra-frame and inter-frame to comprehensively aggregate the informative contexts via local-global \textbf{O}ptimal \textbf{T}ransport (\textbf{AOT}). Specifically, we first establish local- and global-aware token anchors within each frame under the attention guidance, which then optimal transport aggregates the informative contexts from pruned tokens, constructing intra-frame token anchors. Then, building on the temporal frame clips, the first frame within each clip will be considered as the keyframe anchors to ensemble similar information from consecutive frames through optimal transport, while keeping distinct tokens to represent temporal dynamics, leading to efficient token reduction in a training-free manner. Extensive evaluations show that our proposed AOT obtains competitive performances across various short- and long-video benchmarks on leading video LLMs, obtaining substantial computational efficiency while preserving temporal and visual fidelity. Project webpage: \href{this https URL}{AOT}.
Abstract (translated)
视频大型语言模型（VLLMs）展示了强大的视频理解能力，但由于冗余的视觉令牌导致效率低下。现有剪枝方法主要针对帧内空间冗余或在深度较浅的层进行剪枝，导致时空压缩效果不佳，并且未能充分利用长上下文可压缩性。它们经常从合并或裁剪的令牌中丢弃细微但有信息量的上下文。 本文提出了一种新的视角：通过局部-全局最优传输（AOT），详细阐述了帧内和帧间的令牌锚点，从而全面聚合信息上下文。具体而言，我们首先在注意力引导下，在每个视频帧内部建立具有局部和全局意识的令牌锚点，然后通过最优传输从被剪枝的令牌中汇总信息，构建帧内的令牌锚点。接着，在时间片段的基础上，将每个片段中的第一个帧视为关键帧锚点，利用最优传输整合连续帧中的相似信息，并保持区分性标记以表示时间动态变化，从而在无需训练的情况下实现有效的令牌减少。 广泛的评估表明，我们提出的AOT方法在各种短视频和长视频基准测试中，在领先的VLLMs上达到了具有竞争力的性能，同时获得了显著的计算效率并保留了时间和视觉保真度。项目网页：[此处应为实际链接，请访问项目页面了解更多信息](this https URL)。
URL
https://arxiv.org/abs/2603.01400
PDF
-
Unified Vision-Language Modeling via Concept Space Alignment
2026-03-01 13:22:44Yifu Qiu, Paul-Ambroise Duquenne, Holger SchwenkAbstract
We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.
Abstract (translated)
我们介绍了V-SONAR，这是一个从仅文本嵌入空间SONAR（Omnilingual Embeddings Team等，2026年）扩展而来的视觉-语言嵌入空间，支持1500种文字语言和177种语音语言。为了构建V-SONAR，我们提出了一种后处理对齐流程，该流程将现有视觉编码器的表示映射到SONAR空间中。我们全面评估了V-SONAR，并展示了其嵌入在文本到视频检索任务上取得了竞争性性能。配备了OMNISONAR文本解码器，V-SONAR进一步超越了最新的视觉-语言模型，在包括DREAM-1K（BLEU 23.9 vs. 19.6）和PE-VIDEO（BLEU 39.0 vs. 30.0）的视频字幕任务上。 利用V-SONAR，我们首先展示了在SONAR中运行并仅使用英语文本进行训练的大概念模型(LCM；LCM团队等,2024)可以零样本地执行单个和多个视觉概念理解。最后，我们介绍了V-LCM，它将视觉-语言指令微调扩展到了LCM中。V-LCM通过V-SONAR和SONAR将视觉和语言输入编码为统一的潜在嵌入序列，并且它的训练使用了与LCM仅文本预训练相同的潜扩散目标进行下一个嵌入预测。 在大规模多语种和多模态指令微调数据集上的实验突显了V-LCM的潜力：在涵盖图像/视频字幕和问答任务上，V-LCM的表现可媲美最新的视觉-语言模型；而在所有62种测试语言中，其在包含从丰富资源到低资源在内的61种语言中的表现显著超越。
URL
https://arxiv.org/abs/2603.01096
PDF
-
Event-Anchored Frame Selection for Effective Long-Video Understanding
2026-03-01 08:25:37Wang Chen, Yongdong Luo, Yuhui Zeng, Luojun Lin, Tianyu Xie, Fei Chao, Rongrong Ji, Xiawu ZhengAbstract
Massive frame redundancy and limited context window make efficient frame selection crucial for long-video understanding with large vision-language models (LVLMs). Prevailing approaches, however, adopt a flat sampling paradigm which treats the video as an unstructured collection of frames. In this paper, we introduce Event-Anchored Frame Selection (EFS), a hierarchical, event-aware pipeline. Leveraging self-supervised DINO embeddings, EFS first partitions the video stream into visually homogeneous temporal segments, which serve as proxies for semantic events. Within each event, it then selects the most query-relevant frame as an anchor. These anchors act as structural priors that guide a global refinement stage using an adaptive Maximal Marginal Relevance (MMR) scheme. This pipeline ensures the final keyframe set jointly optimizes for event coverage, query relevance, and visual diversity. As a training-free, plug-and-play module, EFS can be seamlessly integrated into off-the-shelf LVLMs, yielding substantial gains on challenging video understanding benchmarks. Specifically, when applied to LLaVA-Video-7B, EFS improves accuracy by 4.7%, 4.9%, and 8.8% on VideoMME, LongVideoBench, and MLVU, respectively.
Abstract (translated)
大规模的帧冗余和有限的上下文窗口使得长视频理解中大型视觉语言模型（LVLM）的有效帧选择变得至关重要。然而，现有方法普遍采用扁平化的采样模式，将视频视为无结构的帧集合。在本文中，我们引入了基于事件的锚点帧选择（EFS），这是一种分层、事件感知的流程。利用自监督DINO嵌入，EFS首先将视频流划分为视觉上同质的时间段，这些时间段充当语义事件的代理。然后，在每个事件内部，它会选择与查询最相关的帧作为锚点。这些锚点作为结构先验引导全局细化阶段使用自适应最大边缘相关性（MMR）方案进行操作。这一流程确保最终的关键帧集合在事件覆盖率、查询相关性和视觉多样性方面共同优化。作为一个无需训练的即插即用模块，EFS可以无缝集成到现成的LVLM中，在具有挑战性的视频理解基准测试上取得了显著的改进。具体而言，当应用于LLaVA-Video-7B时，EFS在VideoMME、LongVideoBench和MLVU上的准确率分别提高了4.7%、4.9%和8.8%。
URL
https://arxiv.org/abs/2603.00983
PDF
-
Wavelet-based Frame Selection by Detecting Semantic Boundary for Long Video Understanding
2026-02-28 07:18:07Wang Chen, Yuhui Zeng, Yongdong Luo, Tianyu Xie, Luojun Lin, Jiayi Ji, Yan Zhang, Xiawu ZhengAbstract
Frame selection is crucial due to high frame redundancy and limited context windows when applying Large Vision-Language Models (LVLMs) to long videos. Current methods typically select frames with high relevance to a given query, resulting in a disjointed set of frames that disregard the narrative structure of video. In this paper, we introduce Wavelet-based Frame Selection by Detecting Semantic Boundary (WFS-SB), a training-free framework that presents a new perspective: effective video understanding hinges not only on high relevance but, more importantly, on capturing semantic shifts - pivotal moments of narrative change that are essential to comprehending the holistic storyline of video. However, direct detection of abrupt changes in the query-frame similarity signal is often unreliable due to high-frequency noise arising from model uncertainty and transient visual variations. To address this, we leverage the wavelet transform, which provides an ideal solution through its multi-resolution analysis in both time and frequency domains. By applying this transform, we decompose the noisy signal into multiple scales and extract a clean semantic change signal from the coarsest scale. We identify the local extrema of this signal as semantic boundaries, which segment the video into coherent clips. Building on this, WFS-SB comprises a two-stage strategy: first, adaptively allocating a frame budget to each clip based on a composite importance score; and second, within each clip, employing the Maximal Marginal Relevance approach to select a diverse yet relevant set of frames. Extensive experiments show that WFS-SB significantly boosts LVLM performance, e.g., improving accuracy by 5.5% on VideoMME, 9.5% on MLVU, and 6.2% on LongVideoBench, consistently outperforming state-of-the-art methods.
Abstract (translated)
帧选择在应用大型视觉-语言模型（LVLM）到长视频时至关重要，因为这可以减少冗余的帧数并适应有限的上下文窗口。当前的方法通常会选择与给定查询高度相关的帧，导致所选帧集合缺乏连贯性，忽略了视频叙事结构的重要性。在这篇论文中，我们提出了基于小波变换检测语义边界（WFS-SB）的训练自由框架，它提供了一个新的视角：有效的视频理解不仅依赖于高相关性，更重要的是捕捉到叙述变化的关键时刻——这些关键时刻对于理解视频的整体故事情节至关重要。 然而，直接从查询-帧相似度信号中检测突变通常会因为模型不确定性以及短暂视觉变化产生的高频噪声而变得不可靠。为了应对这一挑战，我们利用了小波变换，这种方法通过在时间和频率域的多分辨率分析提供了理想解决方案。通过对这种变换的应用，我们将嘈杂信号分解为多个尺度，并从最粗糙尺度中提取出干净的语义变化信号。接着，我们识别该信号中的局部极值作为语义边界，这些边界将视频分割成连贯的片段。 在此基础上，WFS-SB采用了两阶段策略：首先，在每个片段内根据综合重要性分数自适应地分配帧预算；其次，在每一个片段内部应用最大边缘相关方法来选择多样且相关的帧集合。广泛的实验结果表明，WFS-SB显著提升了LVLM的表现力，例如在VideoMME上提高了5.5%的准确性、在MLVU上提高了9.5%，以及在LongVideoBench上提高了6.2%，并且持续优于现有的最佳方法。 该框架通过更加关注视频中叙述结构的变化点，实现了更高效且连贯的帧选择，从而为LVLM处理长视频提供了更为有效的解决方案。
URL
https://arxiv.org/abs/2603.00512
PDF
-
Hierarchical Action Learning for Weakly-Supervised Action Segmentation
2026-02-27 18:48:22Junxian Huang, Ruichu Cai, Hao Zhu, Juntao Fang, Boyan Xu, Weilin Chen, Zijian Li, Shenghua GaoAbstract
Humans perceive actions through key transitions that structure actions across multiple abstraction levels, whereas machines, relying on visual features, tend to over-segment. This highlights the difficulty of enabling hierarchical reasoning in video understanding. Interestingly, we observe that lower-level visual and high-level action latent variables evolve at different rates, with low-level visual variables changing rapidly, while high-level action variables evolve more slowly, making them easier to identify. Building on this insight, we propose the Hierarchical Action Learning (\textbf{HAL}) model for weakly-supervised action segmentation. Our approach introduces a hierarchical causal data generation process, where high-level latent action governs the dynamics of low-level visual features. To model these varying timescales effectively, we introduce deterministic processes to align these latent variables over time. The \textbf{HAL} model employs a hierarchical pyramid transformer to capture both visual features and latent variables, and a sparse transition constraint is applied to enforce the slower dynamics of high-level action variables. This mechanism enhances the identification of these latent variables over time. Under mild assumptions, we prove that these latent action variables are strictly identifiable. Experimental results on several benchmarks show that the \textbf{HAL} model significantly outperforms existing methods for weakly-supervised action segmentation, confirming its practical effectiveness in real-world applications.
Abstract (translated)
人类通过关键转换来感知动作，这些转换跨越多个抽象层次结构化了动作。相比之下，机器依赖视觉特征，往往会过度分割动作片段。这凸显了在视频理解中实现分层推理的难度。有趣的是，我们观察到低级视觉变量和高级动作潜在变量以不同的速度演变：低级视觉变量变化迅速，而高级行动变量则演变得更慢，使其更容易被识别。 基于这一洞察，我们提出了弱监督下的动作分割的层级动作学习（Hierarchical Action Learning, \textbf{HAL}）模型。我们的方法引入了一个分层因果数据生成过程，在此过程中，高层次潜在的动作支配低层次视觉特征的动力学变化。为了有效模拟这些不同的时间尺度，我们在不同时刻间引入了确定性过程来对齐潜在变量。 \textbf{HAL} 模型采用层级金字塔变压器来捕捉视觉特征和潜在变量，并施加稀疏过渡约束以强制执行高级动作变量的缓慢动态演变。此机制增强了长时间内识别这些潜在变量的能力。在适度假设下，我们证明了这些潜在的动作变量具有严格的可辨识性。 实验结果表明，在多个基准测试上，\textbf{HAL} 模型显著优于现有的弱监督动作分割方法，证实其在实际应用中的有效性和实用性。
URL
https://arxiv.org/abs/2602.24275
PDF
-
EgoGraph: Temporal Knowledge Graph for Egocentric Video Understanding
2026-02-27 06:20:58Shitong Sun, Ke Han, Yukai Huang, Weitong Cai, Jifei SongAbstract
Ultra-long egocentric videos spanning multiple days present significant challenges for video understanding. Existing approaches still rely on fragmented local processing and limited temporal modeling, restricting their ability to reason over such extended sequences. To address these limitations, we introduce EgoGraph, a training-free and dynamic knowledge-graph construction framework that explicitly encodes long-term, cross-entity dependencies in egocentric video streams. EgoGraph employs a novel egocentric schema that unifies the extraction and abstraction of core entities, such as people, objects, locations, and events, and structurally reasons about their attributes and interactions, yielding a significantly richer and more coherent semantic representation than traditional clip-based video models. Crucially, we develop a temporal relational modeling strategy that captures temporal dependencies across entities and accumulates stable long-term memory over multiple days, enabling complex temporal reasoning. Extensive experiments on the EgoLifeQA and EgoR1-bench benchmarks demonstrate that EgoGraph achieves state-of-the-art performance on long-term video question answering, validating its effectiveness as a new paradigm for ultra-long egocentric video understanding.
Abstract (translated)
超长的第一人称视频（跨越多天）对视频理解提出了重大挑战。现有的方法仍然依赖于片段化的局部处理和有限的时间建模，限制了它们在如此长时间序列上进行推理的能力。为了解决这些局限性，我们引入了一个名为EgoGraph的框架，这是一个无需训练且动态构建知识图谱的方法，该方法明确地编码了长期跨实体之间的依赖关系。EgoGraph采用了一种新颖的第一人称模式，统一提取和抽象核心实体（如人物、物体、地点和事件），并结构化推理这些实体的属性及互动，从而比传统的基于片段的视频模型提供更丰富且连贯的语义表示。至关重要的是，我们开发了一种时间关系建模策略，捕捉了跨多个实体的时间依赖性，并累积了跨越多天的稳定长期记忆，这使得复杂的时序推理成为可能。 在EgoLifeQA和EgoR1-bench基准测试上的大量实验表明，EgoGraph在长视频问答方面达到了最先进的性能水平，验证了它作为理解超长第一人称视频的新范式的有效性。
URL
https://arxiv.org/abs/2602.23709
PDF
-
LE-NeuS: Latency-Efficient Neuro-Symbolic Video Understanding via Adaptive Temporal Verification
2026-02-26 23:28:13Shawn Liang, Sahil Shah, Chengwei Zhou, SP Sharan, Harsh Goel, Arnab Sanyal, Sandeep Chinchali, Gourav DattaAbstract
Neuro-symbolic approaches to long-form video question answering (LVQA) have demonstrated significant accuracy improvements by grounding temporal reasoning in formal verification. However, existing methods incur prohibitive latency overheads, up to 90x slower than base VLM prompting, rendering them impractical for latency-sensitive edge deployments. We present LE-NeuS, a latency-efficient neuro-symbolic framework that preserves the accuracy benefits of temporal logic-guided video understanding while drastically reducing inference latency. Our key insight is that the dominant computational bottleneck arises from sequential and dense proposition detection across video frames during automaton construction. We address this through two principled optimizations: (1) CLIP guided two-stage adaptive sampling that exploits visual redundancy to skip semantically similar frames while preserving temporal boundaries, and (2) batched proposition detection that parallelizes VLM inference across temporal windows. Theoretically, we derive latency bounds as a function of video length, proposition complexity, and sampling density, establishing conditions under which latency efficiency is achievable. Empirically, on LongVideoBench and Video-MME benchmarks deployed on NVIDIA H100 GPUs, LE-NeuS reduces the latency gap from 90x to approximately 10x while maintaining >10% accuracy gains on temporally complex queries.
Abstract (translated)
神经符号方法在长视频问答（LVQA）中通过将时间推理建立在形式验证基础上，展示了显著的准确性提升。然而，现有的方法由于计算延迟过大而变得不可行，其速度比基础视觉语言模型提示方法慢90倍，这对于对延迟敏感的边缘部署来说是不切实际的。我们提出了LE-NeuS，这是一种低延迟神经符号框架，在保持时间逻辑指导下的视频理解准确性的同时，大大减少了推理延迟。 我们的关键见解在于计算瓶颈主要来自于在构建自动机时对整个视频帧进行顺序和密集的状态检测过程。为此，我们通过两项原理性的优化来解决这一问题：（1）CLIP引导的两阶段自适应采样方法，该方法利用视觉冗余跳过语义相似的帧同时保持时间边界；（2）批量状态检测，将VLM推理并行化处理以跨越多个时间窗口。 理论上，我们推导出了视频长度、状态复杂性和采样密度作为函数影响延迟效率的界限，并建立了实现低延迟计算的有效条件。实验上，在NVIDIA H100 GPU上的LongVideoBench和Video-MME基准测试中，LE-NeuS将延迟差距从90倍减少到大约10倍，同时在处理时间复杂度高的查询时保持超过10%的准确性提升。 这种方法通过优化神经符号模型中的计算瓶颈，使其能够更有效地应用于长视频理解和问答任务，并且对于需要快速响应的应用场景具有重要意义。
URL
https://arxiv.org/abs/2602.23553
PDF
-
Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception
2026-02-26 14:58:59Yiding Sun, Jihua Zhu, Haozhe Cheng, Chaoyi Lu, Zhichuan Yang, Lin Chen, Yaonan WangAbstract
Point cloud video understanding is critical for robotics as it accurately encodes motion and scene interaction. We recognize that 4D datasets are far scarcer than 3D ones, which hampers the scalability of self-supervised 4D models. A promising alternative is to transfer 3D pre-trained models to 4D perception tasks. However, rigorous empirical analysis reveals two critical limitations that impede transfer capability: overfitting and the modality gap. To overcome these challenges, we develop a novel "Align then Adapt" (PointATA) paradigm that decomposes parameter-efficient transfer learning into two sequential stages. Optimal-transport theory is employed to quantify the distributional discrepancy between 3D and 4D datasets, enabling our proposed point align embedder to be trained in Stage 1 to alleviate the underlying modality gap. To mitigate overfitting, an efficient point-video adapter and a spatial-context encoder are integrated into the frozen 3D backbone to enhance temporal modeling capacity in Stage 2. Notably, with the above engineering-oriented designs, PointATA enables a pre-trained 3D model without temporal knowledge to reason about dynamic video content at a smaller parameter cost compared to previous work. Extensive experiments show that PointATA can match or even outperform strong full fine-tuning models, whilst enjoying the advantage of parameter efficiency, e.g. 97.21 \% accuracy on 3D action recognition, $+8.7 \%$ on 4 D action segmentation, and 84.06\% on 4D semantic segmentation.
Abstract (translated)
点云视频理解在机器人技术中至关重要，因为它能准确地编码运动和场景交互。我们认识到，相较于3D数据集而言，4D数据集稀缺得多，这阻碍了自监督4D模型的可扩展性。一种有前景的替代方案是从3D预训练模型迁移至4D感知任务。然而，严格的实证分析揭示出两种关键限制，即过度拟合和模态差距问题，这些问题阻碍了迁移能力。 为了解决这些挑战，我们开发了一种新颖的“对齐后适应”（PointATA）范式，将参数效率高的迁移学习分解成两个连续阶段。利用最优传输理论来量化3D与4D数据集之间的分布差异，使我们的提议点对齐嵌入器能够在第一阶段训练以缓解潜在的模态差距问题。为减轻过度拟合，在第二阶段集成高效的点视频适配器和空间上下文编码器到冻结的3D主干中，从而增强时间建模能力。 值得注意的是，通过上述工程设计，PointATA使一个无时间知识的预训练3D模型能够以较小的参数成本推理出动态视频内容。广泛的实验表明，PointATA可以与强大的全微调模型相匹配甚至超越其性能，同时享受参数效率的优势，例如在3D动作识别中达到97.21%的准确率，在4D动作分割中提高8.7%，以及在4D语义分割中达到84.06%的成绩。
URL
https://arxiv.org/abs/2602.23069
PDF
-
MSJoE: Jointly Evolving MLLM and Sampler for Efficient Long-Form Video Understanding
2026-02-26 12:24:17Wenhui Tan, Xiaoyi Yu, Jiaze Li, Yijing Chen, Jianzhong Ju, Zhenbo Luo, Ruihua Song, Jian LuanAbstract
Efficiently understanding long-form videos remains a fundamental challenge for multimodal large language models (MLLMs). In this paper, we present MLLM-Sampler Joint Evolution (MSJoE), a novel framework that jointly evolves the MLLM and a lightweight key-frame sampler for efficient long-form video understanding. MSJoE builds upon a key assumption that only a small subset of key-frames is truly informative for answering each question to a video. Specifically, MSJoE first reasons out several queries, which describe diverse visual perspectives relevant to the question. Then, these queries interact with a frozen CLIP model to produce a query-frame similarity matrix. Finally, a lightweight sampler predicts key-frame sampling weights from this matrix, selecting a compact set of informative frames, which are then fed into the MLLM for answer generation. Both the MLLM and sampler are jointly optimized through reinforcement learning, enabling co-adaptation of query-reasoning, frame-sampling, and key-frame understanding. A new long-video QA dataset containing 2.8K videos with 7K question-answer pairs is collected to support the training process. Extensive experiments on VideoMME, LongVideoBench, LVBench, and MLVU show that MSJoE achieves 8.0\% accuracy gain upon the base MLLM, and 1.1\% higher accuracy than strongest baseline method.
Abstract (translated)
理解长视频的有效方法仍然是多模态大型语言模型（MLLM）面临的基本挑战。本文介绍了一种新的框架——MLLM-Sampler联合演化（MSJoE），该框架能够使MLLM和一个轻量级的关键帧采样器协同进化，从而实现对长视频的高效理解和分析。 MSJoE基于这样一个关键假设：对于每个问题而言，只有视频中的一小部分关键帧才是真正具有信息价值的。具体来说，MSJoE首先生成几个描述与问题相关且多样化的视觉视角的查询语句。接着，这些查询与一个冻结状态下的CLIP模型互动，以产生一个查询-帧相似度矩阵。然后，轻量级采样器从这个矩阵中预测出关键帧的采样权重，并选择一套精简的信息丰富帧集，这些被选中的帧随后输入到MLLM进行答案生成。 通过强化学习的方式，对MLLM和采样器同时进行优化，使得查询推理、帧采样以及关键帧理解能够共同适应。为此，我们收集了一个新的长视频问答数据集，其中包含2800个视频和7000组问题-答案配对以支持训练过程。 在VideoMME、LongVideoBench、LVBench 和 MLVU上进行的广泛实验表明，MSJoE相比基础MLLM提高了8.0%的准确率，并且比最强基线方法高出1.1%。
URL
https://arxiv.org/abs/2602.22932
PDF
