Video_Caption
-
CoPE-VideoLM: Codec Primitives For Efficient Video Language Models
2026-02-13 18:57:31Sayan Deb Sarkar, R\'emi Pautrat, Ondrej Miksik, Marc Pollefeys, Iro Armeni, Mahdi Rad, Mihai DusmanuAbstract
Video Language Models (VideoLMs) empower AI systems to understand temporal dynamics in videos. To fit to the maximum context window constraint, current methods use keyframe sampling which can miss both macro-level events and micro-level details due to the sparse temporal coverage. Furthermore, processing full images and their tokens for each frame incurs substantial computational overhead. To address these limitations, we propose to leverage video codec primitives (specifically motion vectors and residuals) which natively encode video redundancy and sparsity without requiring expensive full-image encoding for most frames. To this end, we introduce lightweight transformer-based encoders that aggregate codec primitives and align their representations with image encoder embeddings through a pre-training strategy that accelerates convergence during end-to-end fine-tuning. Our approach reduces the time-to-first-token by up to $86\%$ and token usage by up to $93\%$ compared to standard VideoLMs. Moreover, by varying the keyframe and codec primitive densities we are able to maintain or exceed performance on $14$ diverse video understanding benchmarks spanning general question answering, temporal reasoning, long-form understanding, and spatial scene understanding.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13191
PDF
-
Towards Universal Video MLLMs with Attribute-Structured and Quality-Verified Instructions
2026-02-13 15:20:54Yunheng Li, Hengrui Zhang, Meng-Hao Guo, Wenzhao Gao, Shaoyong Jia, Shaohui Jiao, Qibin Hou, Ming-Ming ChengAbstract
Universal video understanding requires modeling fine-grained visual and audio information over time in diverse real-world scenarios. However, the performance of existing models is primarily constrained by video-instruction data that represents complex audiovisual content as single, incomplete descriptions, lacking fine-grained organization and reliable annotation. To address this, we introduce: (i) ASID-1M, an open-source collection of one million structured, fine-grained audiovisual instruction annotations with single- and multi-attribute supervision; (ii) ASID-Verify, a scalable data curation pipeline for annotation, with automatic verification and refinement that enforces semantic and temporal consistency between descriptions and the corresponding audiovisual content; and (iii) ASID-Captioner, a video understanding model trained via Supervised Fine-Tuning (SFT) on the ASID-1M. Experiments across seven benchmarks covering audiovisual captioning, attribute-wise captioning, caption-based QA, and caption-based temporal grounding show that ASID-Captioner improves fine-grained caption quality while reducing hallucinations and improving instruction following. It achieves state-of-the-art performance among open-source models and is competitive with Gemini-3-Pro.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13013
PDF
-
Artic: AI-oriented Real-time Communication for MLLM Video Assistant
2026-02-13 06:00:17Jiangkai Wu, Zhiyuan Ren, Junquan Zhong, Liming Liu, Xinggong ZhangAbstract
AI Video Assistant emerges as a new paradigm for Real-time Communication (RTC), where one peer is a Multimodal Large Language Model (MLLM) deployed in the cloud. This makes interaction between humans and AI more intuitive, akin to chatting with a real person. However, a fundamental mismatch exists between current RTC frameworks and AI Video Assistants, stemming from the drastic shift in Quality of Experience (QoE) and more challenging networks. Measurements on our production prototype also confirm that current RTC fails, causing latency spikes and accuracy drops. To address these challenges, we propose Artic, an AI-oriented RTC framework for MLLM Video Assistants, exploring the shift from "humans watching video" to "AI understanding video." Specifically, Artic proposes: (1) Response Capability-aware Adaptive Bitrate, which utilizes MLLM accuracy saturation to proactively cap bitrate, reserving bandwidth headroom to absorb future fluctuations for latency reduction; (2) Zero-overhead Context-aware Streaming, which allocates limited bitrate to regions most important for the response, maintaining accuracy even under ultra-low bitrates; and (3) Degraded Video Understanding Benchmark, the first benchmark evaluating how RTC-induced video degradation affects MLLM accuracy. Prototype experiments using real-world uplink traces show that compared with existing methods, Artic significantly improves accuracy by 15.12% and reduces latency by 135.31 ms. We will release the benchmark and codes at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2602.12641
PDF
-
MedScope: Incentivizing 'Think with Videos' for Clinical Reasoning via Coarse-to-Fine Tool Calling
2026-02-11 09:47:02Wenjie Li, Yujie Zhang, Haoran Sun, Xingqi He, Hongcheng Gao, Chenglong Ma, Ming Hu, Guankun Wang, Shiyi Yao, Renhao Yang, Hongliang Ren, Lei Wang, Junjun He, Yankai JiangAbstract
Long-form clinical videos are central to visual evidence-based decision-making, with growing importance for applications such as surgical robotics and related settings. However, current multimodal large language models typically process videos with passive sampling or weakly grounded inspection, which limits their ability to iteratively locate, verify, and justify predictions with temporally targeted evidence. To close this gap, we propose MedScope, a tool-using clinical video reasoning model that performs coarse-to-fine evidence seeking over long-form procedures. By interleaving intermediate reasoning with targeted tool calls and verification on retrieved observations, MedScope produces more accurate and trustworthy predictions that are explicitly grounded in temporally localized visual evidence. To address the lack of high-fidelity supervision, we build ClinVideoSuite, an evidence-centric, fine-grained clinical video suite. We then optimize MedScope with Grounding-Aware Group Relative Policy Optimization (GA-GRPO), which directly reinforces tool use with grounding-aligned rewards and evidence-weighted advantages. On full and fine-grained video understanding benchmarks, MedScope achieves state-of-the-art performance in both in-domain and out-of-domain evaluations. Our approach illuminates a path toward medical AI agents that can genuinely "think with videos" through tool-integrated reasoning. We will release our code, models, and data.
Abstract (translated)
URL
https://arxiv.org/abs/2602.13332
PDF
-
VideoSTF: Stress-Testing Output Repetition in Video Large Language Models
2026-02-11 08:40:48Yuxin Cao, Wei Song, Shangzhi Xu, Jingling Xue, Jin Song DongAbstract
Video Large Language Models (VideoLLMs) have recently achieved strong performance in video understanding tasks. However, we identify a previously underexplored generation failure: severe output repetition, where models degenerate into self-reinforcing loops of repeated phrases or sentences. This failure mode is not captured by existing VideoLLM benchmarks, which focus primarily on task accuracy and factual correctness. We introduce VideoSTF, the first framework for systematically measuring and stress-testing output repetition in VideoLLMs. VideoSTF formalizes repetition using three complementary n-gram-based metrics and provides a standardized testbed of 10,000 diverse videos together with a library of controlled temporal transformations. Using VideoSTF, we conduct pervasive testing, temporal stress testing, and adversarial exploitation across 10 advanced VideoLLMs. We find that output repetition is widespread and, critically, highly sensitive to temporal perturbations of video inputs. Moreover, we show that simple temporal transformations can efficiently induce repetitive degeneration in a black-box setting, exposing output repetition as an exploitable security vulnerability. Our results reveal output repetition as a fundamental stability issue in modern VideoLLMs and motivate stability-aware evaluation for video-language systems. Our evaluation code and scripts are available at: this https URL.
Abstract (translated)
最近，视频大型语言模型（Video Large Language Models, VideoLLM）在视频理解任务中表现出色。然而，我们发现了一个此前未被充分探索的生成失败模式：严重的输出重复现象，即模型陷入自我强化的循环，不断重复相同的短语或句子。现有的VideoLLM基准测试主要关注任务准确性和事实正确性，并没有捕捉到这种失效模式。为此，我们提出了VideoSTF（Video Stability Test Framework），这是第一个系统地衡量和压力测试视频生成中输出重复现象的框架。VideoSTF使用三种互补的n-gram基元度量方法来正式定义重复问题，并提供了一个包含10,000个多样化视频的标准测试平台，以及一个控制时间转换的库。 通过利用VideoSTF，我们对十个高级VideoLLM进行了广泛的测试、时序压力测试和对抗性利用。我们的研究发现，输出重复现象普遍存在，并且关键地，它对于输入视频的时间扰动非常敏感。此外，我们还展示了在黑盒设置中，简单的时间转换可以高效地诱导模型出现重复性的退化行为，揭示了这种输出重复模式作为可被利用的安全漏洞。 这些结果表明，输出重复是现代VideoLLM中的一个根本性稳定性问题，并激发了视频-语言系统中以稳定性能为导向的评估需求。我们的评测代码和脚本可在以下链接获取：this https URL.
URL
https://arxiv.org/abs/2602.10639
PDF
-
RoboSubtaskNet: Temporal Sub-task Segmentation for Human-to-Robot Skill Transfer in Real-World Environments
2026-02-10 17:37:35Dharmendra Sharma, Archit Sharma, John Reberio, Vaibhav Kesharwani, Peeyush Thakur, Narendra Kumar Dhar, Laxmidhar BeheraAbstract
Temporally locating and classifying fine-grained sub-task segments in long, untrimmed videos is crucial to safe human-robot collaboration. Unlike generic activity recognition, collaborative manipulation requires sub-task labels that are directly robot-executable. We present RoboSubtaskNet, a multi-stage human-to-robot sub-task segmentation framework that couples attention-enhanced I3D features (RGB plus optical flow) with a modified MS-TCN employing a Fibonacci dilation schedule to capture better short-horizon transitions such as reach-pick-place. The network is trained with a composite objective comprising cross-entropy and temporal regularizers (truncated MSE and a transition-aware term) to reduce over-segmentation and to encourage valid sub-task progressions. To close the gap between vision benchmarks and control, we introduce RoboSubtask, a dataset of healthcare and industrial demonstrations annotated at the sub-task level and designed for deterministic mapping to manipulator primitives. Empirically, RoboSubtaskNet outperforms MS-TCN and MS-TCN++ on GTEA and our RoboSubtask benchmark (boundary-sensitive and sequence metrics), while remaining competitive on the long-horizon Breakfast benchmark. Specifically, RoboSubtaskNet attains F1 @ 50 = 79.5%, Edit = 88.6%, Acc = 78.9% on GTEA; F1 @ 50 = 30.4%, Edit = 52.0%, Acc = 53.5% on Breakfast; and F1 @ 50 = 94.2%, Edit = 95.6%, Acc = 92.2% on RoboSubtask. We further validate the full perception-to-execution pipeline on a 7-DoF Kinova Gen3 manipulator, achieving reliable end-to-end behavior in physical trials (overall task success approx 91.25%). These results demonstrate a practical path from sub-task level video understanding to deployed robotic manipulation in real-world settings.
Abstract (translated)
在长时间未剪辑的视频中定位和分类精细粒度的任务子段对于安全的人机协作至关重要。与通用活动识别不同，协作操作需要可以直接由机器人执行的任务子标签。我们提出了一种多阶段的人类到机器人的任务分割框架RoboSubtaskNet，该框架结合了增强注意力机制的I3D特征（RGB和光流）以及采用斐波那契膨胀时间表修改后的MS-TCN架构，以更好地捕捉如“伸手-抓取-放置”这类短时域内的转换。网络通过一个包含交叉熵和时间正则化器（截断MSE和过渡感知项）的复合目标进行训练，以减少过度分割并鼓励有效的子任务进展。 为了弥合视觉基准与控制之间的差距，我们引入了RoboSubtask数据集，该数据集包含了医疗保健和工业演示，并在子任务级别进行了注释，旨在确定性地映射到机械臂的基本操作。经验表明，在GTEA和我们的RoboSubtask基准测试（边界敏感性和序列度量）上，RoboSubtaskNet的表现优于MS-TCN和MS-TCN++，同时在长时域的Breakfast数据集上也具有竞争力。具体而言，RoboSubtaskNet在GTEA上的F1@50 = 79.5%，Edit = 88.6%，Acc = 78.9%；在Breakfast上的F1@50 = 30.4%，Edit = 52.0%，Acc = 53.5%；以及在RoboSubtask上的F1@50 = 94.2%，Edit = 95.6%，Acc = 92.2%。我们进一步在一个7自由度的Kinova Gen3机械臂上验证了整个感知到执行的管道，在物理试验中实现了可靠的整体端到端行为（总体任务成功率约为91.25%）。这些结果展示了从子任务级别视频理解到现实世界环境中部署机器人操作的实际路径。 上述内容翻译自原文，详细描述了RoboSubtaskNet框架及其在不同数据集上的性能表现。
URL
https://arxiv.org/abs/2602.10015
PDF
-
SemanticMoments: Training-Free Motion Similarity via Third Moment Features
2026-02-09 19:47:56Saar Huberman, Kfir Goldberg, Or Patashnik, Sagie Benaim, Ron MokadyAbstract
Retrieving videos based on semantic motion is a fundamental, yet unsolved, problem. Existing video representation approaches overly rely on static appearance and scene context rather than motion dynamics, a bias inherited from their training data and objectives. Conversely, traditional motion-centric inputs like optical flow lack the semantic grounding needed to understand high-level motion. To demonstrate this inherent bias, we introduce the SimMotion benchmarks, combining controlled synthetic data with a new human-annotated real-world dataset. We show that existing models perform poorly on these benchmarks, often failing to disentangle motion from appearance. To address this gap, we propose SemanticMoments, a simple, training-free method that computes temporal statistics (specifically, higher-order moments) over features from pre-trained semantic models. Across our benchmarks, SemanticMoments consistently outperforms existing RGB, flow, and text-supervised methods. This demonstrates that temporal statistics in a semantic feature space provide a scalable and perceptually grounded foundation for motion-centric video understanding.
Abstract (translated)
基于语义动作检索视频是一项基本但尚未解决的问题。现有的视频表示方法过于依赖静态外观和场景上下文，而忽视了动态动作，这种偏见源自它们的训练数据和目标设定。相比之下，传统的以动作为中心的输入（如光流）缺乏理解高层次动作所需的语义背景。为了展示这一内在偏见，我们引入了SimMotion基准测试集，它结合了受控合成数据与新的、由人类标注的真实世界数据集。我们发现现有模型在这些基准上表现不佳，经常无法将运动和外观分开。 为了解决这个差距，我们提出了SemanticMoments，这是一种简单且无需训练的方法，通过对预训练语义模型的特征计算时间统计（特别是高阶矩）。在我们的基准测试中，SemanticMoments始终优于现有的RGB、光流以及文本监督方法。这表明，在语义特征空间中的时间统计数据为以动作为中心的视频理解提供了一个可扩展且感知基础良好的框架。
URL
https://arxiv.org/abs/2602.09146
PDF
-
Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionals
2026-02-09 16:43:37Puqi Zhou (George Mason University, Fairfax, VA, USA), Ali Asgarov (Virginia Tech, Blacksburg, VA, USA), Aafiya Hussain (Virginia Tech, Blacksburg, VA, USA), Wonjoon Park (University of Maryland, College Park, MD, USA), Amit Paudyal (George Mason University, Fairfax, VA, USA), Sameep Shrestha (George Mason University, Fairfax, VA, USA), Chia-wei Tang (Virginia Tech, Blacksburg, VA, USA), Michael F. Lighthiser (George Mason University, Fairfax, VA, USA), Michael R. Hieb (George Mason University, Fairfax, VA, USA), Xuesu Xiao (George Mason University, Fairfax, VA, USA), Chris Thomas (Virginia Tech, Blacksburg, VA, USA), Sungsoo Ray Hong (George Mason University, Fairfax, VA, USA)Abstract
Videos from fleets of ground robots can advance public safety by providing scalable situational awareness and reducing professionals' burden. Yet little is known about how to design and integrate multi-robot videos into public safety workflows. Collaborating with six police agencies, we examined how such videos could be made practical. In Study 1, we presented the first testbed for multi-robot ground video sensemaking. The testbed includes 38 events-of-interest (EoI) relevant to public safety, a dataset of 20 robot patrol videos (10 day/night pairs) covering EoI types, and 6 design requirements aimed at improving current video sensemaking practices. In Study 2, we built MRVS, a tool that augments multi-robot patrol video streams with a prompt-engineered video understanding model. Participants reported reduced manual workload and greater confidence with LLM-based explanations, while noting concerns about false alarms and privacy. We conclude with implications for designing future multi-robot video sensemaking tools. The testbed is available at this https URL\_VideoSensemaking
Abstract (translated)
地面机器人车队的视频可以通过提供可扩展的情境感知和减轻专业人员负担来提升公共安全。然而，关于如何设计并整合多机器人视频到公共安全管理流程中的知识还很少。与六家警察机构合作，我们研究了此类视频如何变得实用。 在第一项研究中，我们展示了第一个用于多机器人地面视频理解的测试平台。该测试平台包括38个与公共安全相关的事件（EoI），以及一个包含20段巡逻机器人视频的数据集（10组昼夜视频），这些数据涵盖了各种类型的事件，并确定了6个旨在改进当前视频理解和分析实践的设计要求。 在第二项研究中，我们构建了MRVS工具，该工具使用经过提示工程优化的视频理解模型来增强多机器人的巡逻视频流。参与者报告说，基于LLM（大型语言模型）解释的帮助减少了手动工作量，并且提高了他们的信心，同时他们也表达了对误报和隐私问题的关注。 我们的研究总结指出，在设计未来的多机器人视频理解和分析工具时，应当考虑这些发现的启示作用。测试平台可以在以下链接找到：https://this.is.URL\_VideoSensemaking （注：上述URL只是一个占位符，请用实际提供的有效网址替换）
URL
https://arxiv.org/abs/2602.08882
PDF
-
TiFRe: Text-guided Video Frame Reduction for Efficient Video Multi-modal Large Language Models
2026-02-09 16:24:53Xiangtian Zheng, Zishuo Wang, Yuxin PengAbstract
With the rapid development of Large Language Models (LLMs), Video Multi-Modal Large Language Models (Video MLLMs) have achieved remarkable performance in video-language tasks such as video understanding and question answering. However, Video MLLMs face high computational costs, particularly in processing numerous video frames as input, which leads to significant attention computation overhead. A straightforward approach to reduce computational costs is to decrease the number of input video frames. However, simply selecting key frames at a fixed frame rate (FPS) often overlooks valuable information in non-key frames, resulting in notable performance degradation. To address this, we propose Text-guided Video Frame Reduction (TiFRe), a framework that reduces input frames while preserving essential video information. TiFRe uses a Text-guided Frame Sampling (TFS) strategy to select key frames based on user input, which is processed by an LLM to generate a CLIP-style prompt. Pre-trained CLIP encoders calculate the semantic similarity between the prompt and each frame, selecting the most relevant frames as key frames. To preserve video semantics, TiFRe employs a Frame Matching and Merging (FMM) mechanism, which integrates non-key frame information into the selected key frames, minimizing information loss. Experiments show that TiFRe effectively reduces computational costs while improving performance on video-language tasks.
Abstract (translated)
随着大型语言模型（LLMs）的快速发展，视频多模态大型语言模型（Video MLLMs）在视频理解和问答等任务中取得了显著的成绩。然而，Video MLLMs面临着高昂的计算成本问题，特别是在处理大量视频帧输入时会导致显著的关注力计算开销。减少计算成本的一种直接方法是降低输入视频帧的数量。但是，简单地以固定帧率（FPS）选择关键帧往往会忽略非关键帧中的重要信息，导致性能下降。 为了应对这一挑战，我们提出了文本引导的视频帧缩减框架（Text-guided Video Frame Reduction, TiFRe），该框架在减少输入帧数量的同时保留了重要的视频信息。TiFRe采用了一种基于用户输入的文本引导的帧采样策略（TFS）来选择关键帧，并由LLM生成类似CLIP风格的提示词。预训练好的CLIP编码器计算这些提示词与每个视频帧之间的语义相似度，从而选出最相关的帧作为关键帧。 为了保持视频的语义信息不丢失，TiFRe采用了一个帧匹配和合并（Frame Matching and Merging, FMM）机制，将非关键帧的信息整合到选定的关键帧中，从而最小化信息损失。实验结果表明，TiFRe不仅能有效降低计算成本，还能提升在视频语言任务中的性能表现。
URL
https://arxiv.org/abs/2602.08861
PDF
-
OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence
2026-02-09 14:06:17Feilong Tang, Xiang An, Yunyao Yan, Yin Xie, Bin Qin, Kaicheng Yang, Yifei Shen, Yuanhan Zhang, Chunyuan Li, Shikun Feng, Changrui Chen, Huajie Tan, Ming Hu, Manyuan Zhang, Bo Li, Ziyong Feng, Ziwei Liu, Zongyuan Ge, Jiankang DengAbstract
Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., Codecs. Method. OneVision-Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OV-Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision-Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics. Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. When integrated into LLM, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OV-Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Codec-aligned, patch-level sparsity is a foundational principle, enabling OV-Encoder as a scalable engine for next-generation visual generalists.
Abstract (translated)
假设。人工智能的通用智能在根本上是一个压缩问题。有效的压缩需要共振：深度学习在架构与数据的基本结构相匹配时表现最佳。这些是基本原则。然而，现代视觉架构已经偏离了这一真理：视觉信号高度冗余，而判别信息（即惊喜）则非常稀疏。当前模型对密集的像素网格进行均匀处理，浪费了大量的计算资源来处理静态背景，而不是专注于定义运动和意义的预测残差上。我们认为，要解决视觉理解问题，我们必须使架构与视频的信息论原理——编码器-解码器系统（Codec）相一致。 方法。OneVision-Encoder 通过将预测性的视觉结构压缩为语义含义来对视频进行编码。通过采用 Codec 补丁化技术，OV-Encoder 放弃了均匀计算，并专注于信号熵丰富的区域，这些区域占总区域的3.1%-25%之间。为了在不规则令牌布局下统一空间和时间推理，OneVision-Encoder 使用共享的三维RoPE（旋转位置嵌入），并通过超过一百万个语义概念的大规模集群判别目标进行训练，同时捕捉对象持久性和运动动态。 证据。结果验证了我们的核心假设：效率与准确度并不是权衡的关系；它们是正相关的。当集成到大型语言模型中时，OneVision-Encoder 在16个图像、视频和文档理解基准测试上始终优于强大的视觉骨干网络，如Qwen3-ViT和SigLIP2，尽管使用了更少的视觉令牌和预训练数据。值得注意的是，在视频理解任务中，OV-Encoder 相对于 Qwen3-ViT 平均提高了4.1%。 Codec 对齐、补丁级稀疏性是基本原则，使 OV-Encoder 成为下一代视觉通用智能的基础引擎。
URL
https://arxiv.org/abs/2602.08683
PDF
-
ALIVE: Animate Your World with Lifelike Audio-Video Generation
2026-02-09 14:06:03Ying Guo, Qijun Gan, Yifu Zhang, Jinlai Liu, Yifei Hu, Pan Xie, Dongjun Qian, Yu Zhang, Ruiqi Li, Yuqi Zhang, Ruibiao Lu, Xiaofeng Mei, Bo Han, Xiang Yin, Bingyue Peng, Zehuan YuanAbstract
Video generation is rapidly evolving towards unified audio-video generation. In this paper, we present ALIVE, a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation. In particular, the model unlocks the Text-to-Video&Audio (T2VA) and Reference-to-Video&Audio (animation) capabilities compared to the T2V foundation models. To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch which includes TA-CrossAttn for temporally-aligned cross-modal fusion and UniTemp-RoPE for precise audio-visual alignment. Meanwhile, a comprehensive data pipeline consisting of audio-video captioning, quality control, etc., is carefully designed to collect high-quality finetuning data. Additionally, we introduce a new benchmark to perform a comprehensive model test and comparison. After continue pretraining and finetuning on million-level high-quality data, ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions. With detailed recipes and benchmarks, we hope ALIVE helps the community develop audio-video generation models more efficiently. Official page: this https URL.
Abstract (translated)
视频生成正迅速发展为统一的音视频生成。在本文中，我们介绍了ALIVE模型，这是一个将预训练的文本到视频（Text-to-Video，T2V）模型适应于Sora风格的音视频生成和动画工作的生成模型。特别地，与基础的T2V模型相比，该模型解锁了文本到视频音频（Text-to-Video&Audio, T2VA）以及参考到视频音频（动画）的能力。为了支持视听同步及参考动画，我们增强了流行的MMDiT架构，加入了一个联合音视频分支，其中包括用于时间对齐的跨模态融合技术TA-CrossAttn和用于精确视听对齐的UniTemp-RoPE。 同时，设计了一套全面的数据处理流程，包括音频-视频配字幕、质量控制等步骤，以收集高质量的微调数据。此外，我们还引入了一个新的基准测试来进行全面的模型性能测试与比较。经过百万级高质数据的持续预训练和微调之后，ALIVE展现了卓越的性能，在多个方面超过了开源模型，并且达到了或超过了业内最先进的商业解决方案。 通过详细的实现步骤和基准测试标准，希望ALIVE能帮助社区更高效地开发音视频生成模型。官方页面: [提供链接]
URL
https://arxiv.org/abs/2602.08682
PDF
-
Vista: Scene-Aware Optimization for Streaming Video Question Answering under Post-Hoc Queries
2026-02-09 10:00:22Haocheng Lu, Nan Zhang, Wei Tao, Xiaoyang Qu, Guokuan Li, Jiguang Wan, Jianzong WangAbstract
Streaming video question answering (Streaming Video QA) poses distinct challenges for multimodal large language models (MLLMs), as video frames arrive sequentially and user queries can be issued at arbitrary time points. Existing solutions relying on fixed-size memory or naive compression often suffer from context loss or memory overflow, limiting their effectiveness in long-form, real-time scenarios. We present Vista, a novel framework for scene-aware streaming video QA that enables efficient and scalable reasoning over continuous video streams. The innovation of Vista can be summarized in three aspects: (1) scene-aware segmentation, where Vista dynamically clusters incoming frames into temporally and visually coherent scene units; (2) scene-aware compression, where each scene is compressed into a compact token representation and stored in GPU memory for efficient index-based retrieval, while full-resolution frames are offloaded to CPU memory; and (3) scene-aware recall, where relevant scenes are selectively recalled and reintegrated into the model input upon receiving a query, enabling both efficiency and completeness. Vista is model-agnostic and integrates seamlessly with a variety of vision-language backbones, enabling long-context reasoning without compromising latency or memory efficiency. Extensive experiments on StreamingBench demonstrate that Vista achieves state-of-the-art performance, establishing a strong baseline for real-world streaming video understanding.
Abstract (translated)
流媒体视频问答（Streaming Video QA）对多模态大型语言模型（Multimodal Large Language Models，MLLMs）提出了独特的挑战。由于视频帧是按顺序到达的，并且用户可以在任意时间点提出问题，因此现有的解决方案依赖于固定大小的记忆或简单压缩方法往往会导致上下文丢失或内存溢出，在长篇幅、实时场景中效果受限。我们介绍了一个名为Vista的新框架，该框架专门用于基于场景感知的流媒体视频问答，使对连续视频流的有效和可扩展推理成为可能。 Vista 的创新之处可以总结为三个方面： 1. **基于场景的分割**：Vista 动态地将传入的帧聚类成时间和视觉上连贯的场景单元。 2. **基于场景的压缩**：每个场景都被压缩成一个紧凑的令牌表示并存储在 GPU 内存中，以实现高效的索引检索。同时，全分辨率帧被卸载到 CPU 内存。 3. **基于场景的召回**：在接受到查询时，相关场景会被选择性地召回并重新整合到模型输入中，从而保证效率和完整性。 Vista 是一种与多种视觉语言骨干网无缝集成、无需牺牲延迟或内存效率就能实现长上下文推理的方法。在 StreamingBench 上进行的广泛实验表明，Vista 达到了最先进的性能，并为实际流媒体视频理解建立了强有力的基准。
URL
https://arxiv.org/abs/2602.08448
PDF
-
Demo-ICL: In-Context Learning for Procedural Video Knowledge Acquisition
2026-02-09 09:51:29Yuhao Dong, Shulin Tian, Shuai Liu, Shuangrui Ding, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Ziwei LiuAbstract
Despite the growing video understanding capabilities of recent Multimodal Large Language Models (MLLMs), existing video benchmarks primarily assess understanding based on models' static, internal knowledge, rather than their ability to learn and adapt from dynamic, novel contexts from few examples. To bridge this gap, we present Demo-driven Video In-Context Learning, a novel task focused on learning from in-context demonstrations to answer questions about the target videos. Alongside this, we propose Demo-ICL-Bench, a challenging benchmark designed to evaluate demo-driven video in-context learning capabilities. Demo-ICL-Bench is constructed from 1200 instructional YouTube videos with associated questions, from which two types of demonstrations are derived: (i) summarizing video subtitles for text demonstration; and (ii) corresponding instructional videos as video demonstrations. To effectively tackle this new challenge, we develop Demo-ICL, an MLLM with a two-stage training strategy: video-supervised fine-tuning and information-assisted direct preference optimization, jointly enhancing the model's ability to learn from in-context examples. Extensive experiments with state-of-the-art MLLMs confirm the difficulty of Demo-ICL-Bench, demonstrate the effectiveness of Demo-ICL, and thereby unveil future research directions.
Abstract (translated)
尽管最近的多模态大型语言模型（MLLMs）在视频理解能力方面有所提升，但现有的视频基准测试主要基于评估模型的静态、内部知识，而不是它们从少量动态新上下文中学习和适应的能力。为了填补这一空白，我们提出了由演示驱动的视频情境学习任务，该任务专注于通过在情境中的演示来学习并回答关于目标视频的问题。与此相伴，我们还提出了一项新的挑战性基准——Demo-ICL-Bench，旨在评估由演示驱动的视频情境学习能力。 Demo-ICL-Bench 由1200个带有相关问题的教育类YouTube视频构建而成，从中派生出两种类型的示范：(i) 摘要视频字幕以生成文本示例；和(ii) 对应的教程视频作为视频示例。为了有效应对这一新挑战，我们开发了Demo-ICL，这是一种具有两阶段训练策略的MLLM：视频监督微调和信息辅助直接偏好优化，共同增强了模型从情境中例子学习的能力。 与最先进的多模态大型语言模型进行广泛实验后证实了Demo-ICL-Bench 的难度，并展示了 Demo-ICL 的有效性，从而揭示了未来的研究方向。
URL
https://arxiv.org/abs/2602.08439
PDF
-
E-VAds: An E-commerce Short Videos Understanding Benchmark for MLLMs
2026-02-09 07:43:38Xianjie Liu, Yiman Hu, Liang Wu, Ping Hu, Yixiong Zou, Jian Xu, Bo ZhengAbstract
E-commerce short videos represent a high-revenue segment of the online video industry characterized by a goal-driven format and dense multi-modal signals. Current models often struggle with these videos because existing benchmarks focus primarily on general-purpose tasks and neglect the reasoning of commercial intent. In this work, we first propose a \textbf{multi-modal information density assessment framework} to quantify the complexity of this domain. Our evaluation reveals that e-commerce content exhibits substantially higher density across visual, audio, and textual modalities compared to mainstream datasets, establishing a more challenging frontier for video understanding. To address this gap, we introduce \textbf{E-commerce Video Ads Benchmark (E-VAds)}, which is the first benchmark specifically designed for e-commerce short video understanding. We curated 3,961 high-quality videos from Taobao covering a wide range of product categories and used a multi-agent system to generate 19,785 open-ended Q&A pairs. These questions are organized into two primary dimensions, namely Perception and Cognition and Reasoning, which consist of five distinct tasks. Finally, we develop \textbf{E-VAds-R1}, an RL-based reasoning model featuring a multi-grained reward design called \textbf{MG-GRPO}. This strategy provides smooth guidance for early exploration while creating a non-linear incentive for expert-level precision. Experimental results demonstrate that E-VAds-R1 achieves a 109.2% performance gain in commercial intent reasoning with only a few hundred training samples.
Abstract (translated)
电子商务短视频代表了在线视频行业中高收入的一部分，其特点是目标导向的格式和密集多模态信号。目前的模型在处理这些视频时常常遇到困难，因为现有的基准测试主要关注通用任务，并忽视了商业意图推理。在这项工作中，我们首先提出了一种**多模态信息密度评估框架**来量化该领域的复杂性。我们的评估表明，与主流数据集相比，电子商务内容在视觉、音频和文本模式上表现出更高的密集度，为视频理解建立了一个更具挑战性的前沿领域。 为了弥补这一差距，我们引入了**电子商务视频广告基准（E-VAds）**，这是首个专门针对电子商务短视频理解设计的基准测试。我们从淘宝收集了3,961条高质量视频，涵盖了广泛的商品类别，并使用多代理系统生成了19,785对开放式问答。这些问题按照感知和认知及推理两个主要维度组织，并包括五个不同的任务。 最后，我们开发了**E-VAds-R1**，这是一个基于强化学习（RL）的推理模型，具有称为**MG-GRPO**的多粒度奖励设计策略。这种策略为早期探索提供了平滑指导，同时创造了一个非线性的激励机制以达到专家级精度水平。实验结果表明，在只有几百个训练样本的情况下，E-VAds-R1在商业意图推理上实现了109.2%的性能提升。
URL
https://arxiv.org/abs/2602.08355
PDF
-
VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
2026-02-08 19:39:32Issar Tzachor, Dvir Samuel, Rami Ben-AriAbstract
Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for vision tasks, typically through fine-tuning to produce universal representations. However, their performance on video remains inferior to Video Foundation Models (VFMs). In this paper, we focus on leveraging MLLMs for video-text embedding and retrieval. We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information. Leveraging this insight, we demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training. Building on these findings, we introduce a lightweight text-based alignment strategy which maps dense video captions to short summaries and enables task-related video-text embedding learning without visual supervision. Remarkably, without any fine-tuning beyond text, our method outperforms current methods, often by a substantial margin, achieving state-of-the-art results across common video retrieval benchmarks.
Abstract (translated)
近期的研究将生成型多模态大规模语言模型（MLLM）适应为视觉任务的嵌入抽取器，通常通过微调来产生通用表示。然而，在视频处理方面，它们的表现仍不及视频基础模型（VFMs）。本文专注于利用MLLM进行视频-文本嵌入和检索。 首先，我们进行了系统的逐层分析，显示中间层的预训练MLLM已经编码了大量与任务相关的信息。基于这一发现，我们展示了将中间层嵌入与校准后的MLLM头相结合，在无需任何训练的情况下就能实现强大的零样本检索性能。 在此基础上，我们提出了一种轻量级的文本对齐策略，该策略可以将密集视频描述映射到简短摘要，并在没有视觉监督的情况下学习任务相关的视频-文本嵌入。值得注意的是，除了文本外不进行任何微调，我们的方法超越了现有方法，在多个常见的视频检索基准测试中取得了最先进的结果。
URL
https://arxiv.org/abs/2602.08099
PDF
-
FlashVID: Efficient Video Large Language Models via Training-free Tree-based Spatiotemporal Token Merging
2026-02-08 15:56:46Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, Zhuotao TianAbstract
Although Video Large Language Models (VLLMs) have shown remarkable capabilities in video understanding, they are required to process high volumes of visual tokens, causing significant computational inefficiency. Existing VLLMs acceleration frameworks usually compress spatial and temporal redundancy independently, which overlooks the spatiotemporal relationships, thereby leading to suboptimal spatiotemporal compression. The highly correlated visual features are likely to change in spatial position, scale, orientation, and other attributes over time due to the dynamic nature of video. Building on this insight, we introduce FlashVID, a training-free inference acceleration framework for VLLMs. Specifically, FlashVID utilizes Attention and Diversity-based Token Selection (ADTS) to select the most representative tokens for basic video representation, then applies Tree-based Spatiotemporal Token Merging (TSTM) for fine-grained spatiotemporal redundancy elimination. Extensive experiments conducted on three representative VLLMs across five video understanding benchmarks demonstrate the effectiveness and generalization of our method. Notably, by retaining only 10% of visual tokens, FlashVID preserves 99.1% of the performance of LLaVA-OneVision. Consequently, FlashVID can serve as a training-free and plug-and-play module for extending long video frames, which enables a 10x increase in video frame input to Qwen2.5-VL, resulting in a relative improvement of 8.6% within the same computational budget. Code is available at this https URL.
Abstract (translated)
尽管视频大型语言模型（VLLMs）在视频理解方面表现出色，但它们处理大量视觉令牌的需求导致了显著的计算效率低下。现有的VLLM加速框架通常独立地压缩空间和时间冗余，这忽视了时空关系，从而导致次优的时空压缩效果。由于视频的动态特性，高度相关的视觉特征可能随时间在空间位置、尺度、方向等属性上发生变化。 基于这一见解，我们引入了FlashVID，这是一个用于VLLM推理加速且无需训练框架。具体而言，FlashVID利用注意力和多样性令牌选择（ADTS）来选取代表性的视频基本表示的令牌，然后应用树基时空令牌合并（TSTM）进行细粒度的时空冗余消除。 在五个视频理解基准测试中的三个代表性VLLM上进行的广泛实验表明了我们方法的有效性和泛化能力。值得注意的是，通过保留仅10%的视觉令牌，FlashVID能够保持LLaVA-OneVision 99.1%的表现水平。因此，FlashVID可以作为一个无需训练且即插即用的模块来扩展长视频帧，从而在不增加计算预算的情况下，使Qwen2.5-VL的视频帧输入量增加了10倍，并相对提高了8.6%的表现。 代码可在[此链接](https://this-url.com)获得。
URL
https://arxiv.org/abs/2602.08024
PDF
-
D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioning
2026-02-08 13:16:33Changli Tang, Tianyi Wang, Fengyun Rao, Jing Lyu, Chao ZhangAbstract
Spoken dialogue is a primary source of information in videos; therefore, accurately identifying who spoke what and when is essential for deep video understanding. We introduce D-ORCA, a \textbf{d}ialogue-centric \textbf{o}mni-modal large language model optimized for \textbf{r}obust audio-visual \textbf{ca}ptioning. We further curate DVD, a large-scale, high-quality bilingual dataset comprising nearly 40,000 multi-party dialogue videos for training and 2000 videos for evaluation in English and Mandarin, addressing a critical gap in the open-source ecosystem. To ensure fine-grained captioning accuracy, we adopt group relative policy optimization with three novel reward functions that assess speaker attribution accuracy, global speech content accuracy, and sentence-level temporal boundary alignment. These rewards are derived from evaluation metrics widely used in speech processing and, to our knowledge, are applied for the first time as reinforcement learning objectives for audio-visual captioning. Extensive experiments demonstrate that D-ORCA substantially outperforms existing open-source models in speaker identification, speech recognition, and temporal grounding. Notably, despite having only 8 billion parameters, D-ORCA achieves performance competitive with Qwen3-Omni across several general-purpose audio-visual understanding benchmarks. Demos are available at \href{this https URL}{this https URL}. Our code, data, and checkpoints will be available at \href{this https URL}{this https URL}.
Abstract (translated)
口头对话是视频中的主要信息来源，因此准确地识别谁说了什么和什么时候说的对于深入理解视频内容至关重要。我们介绍了D-ORCA，这是一种以对话为中心、面向多模态的大规模语言模型，专门用于鲁棒性的音频-视觉字幕生成（**d**ialogue-centric **o**mni-modal large language model optimized for **r**obust audio-visual **c**aptioning）。此外，我们还策划了DVD数据集，这是一个大规模、高质量的双语数据集，包含近40,000个多方对话视频用于训练以及2000个视频用于评估，这些视频覆盖了英语和普通话。这一数据集填补了开源生态系统中的一个重要空白。 为了确保细粒度字幕的准确性，我们采用了一种基于群体相对策略优化的方法，并引入了三个新颖的奖励函数来评价说话者属性准确率、全球语义内容准确率以及句子级别的时间边界对齐精度。这些奖励函数源自广泛使用的语音处理评估指标，在我们的知识范围内，它们首次被用作音频-视觉字幕生成中的强化学习目标。 通过广泛的实验验证，我们发现D-ORCA在说话人识别、语音识别和时序定位方面大幅优于现有的开源模型。值得注意的是，尽管只有80亿个参数，D-ORCA在多项通用的音频-视频理解基准测试中表现出与Qwen3-Omni相当的性能。 演示可以访问此链接：[this https URL]（请将括号中的内容替换为实际URL）。 我们的代码、数据和检查点将在以下网址提供：[this https URL]（同样，请将括号中的内容替换为实际URL）。
URL
https://arxiv.org/abs/2602.07960
PDF
-
VideoTemp-o3: Harmonizing Temporal Grounding and Video Understanding in Agentic Thinking-with-Videos
2026-02-08 03:45:50Wenqi Liu, Yunxiao Wang, Shijie Ma, Meng Liu, Qile Su, Tianke Zhang, Haonan Fan, Changyi Liu, Kaiyu Jiang, Jiankang Chen, Kaiyu Tang, Bin Wen, Fan Yang, Tingting Gao, Han Li, Yinwei Wei, Xuemeng SongAbstract
In long-video understanding, conventional uniform frame sampling often fails to capture key visual evidence, leading to degraded performance and increased hallucinations. To address this, recent agentic thinking-with-videos paradigms have emerged, adopting a localize-clip-answer pipeline in which the model actively identifies relevant video segments, performs dense sampling within those clips, and then produces answers. However, existing methods remain inefficient, suffer from weak localization, and adhere to rigid workflows. To solve these issues, we propose VideoTemp-o3, a unified agentic thinking-with-videos framework that jointly models video grounding and question answering. VideoTemp-o3 exhibits strong localization capability, supports on-demand clipping, and can refine inaccurate localizations. Specifically, in the supervised fine-tuning stage, we design a unified masking mechanism that encourages exploration while preventing noise. For reinforcement learning, we introduce dedicated rewards to mitigate reward hacking. Besides, from the data perspective, we develop an effective pipeline to construct high-quality long video grounded QA data, along with a corresponding benchmark for systematic evaluation across various video durations. Experimental results demonstrate that our method achieves remarkable performance on both long video understanding and grounding.
Abstract (translated)
在长视频理解领域，传统的均匀帧采样方法常常无法捕捉到关键的视觉证据，从而导致性能下降和幻觉增多（即系统生成不符合事实的答案）。为了解决这一问题，最近出现了一些基于代理思考与视频处理的新范式，这些方法采用了一种“定位-剪辑-回答”的工作流程，在该流程中，模型主动识别相关的视频片段，并在这些片段内进行密集采样，最后给出答案。然而，现有的方法仍然存在效率低下、定位能力弱以及依赖固定的工作流程等问题。 为了应对这些问题，我们提出了VideoTemp-o3，这是一种统一的代理视频思考框架，能够同时建模视频基础设定（即确定视频中哪些部分与问题相关）和问答任务。VideoTemp-o3展现了强大的定位能力，并支持按需剪辑，还可以修正不准确的定位结果。具体而言，在监督微调阶段，我们设计了一种统一的掩码机制以鼓励探索并防止噪声干扰。在强化学习方面，我们引入了专门的奖励机制来减轻奖励篡改的问题。 此外，从数据角度来看，我们开发了一个有效的流程来构建高质量的长视频基础设定问答数据集，并且建立了一个对应的基准测试系统，用于评估不同长度视频的综合性能。 实验结果表明，我们的方法在长视频理解和基础设定任务上都取得了显著的效果。
URL
https://arxiv.org/abs/2602.07801
PDF
-
Process-of-Thought Reasoning for Videos
2026-02-07 20:25:46Jusheng Zhang, Kaitong Cai, Jian Wang, Yongsen Zheng, Kwok-Yan Lam, Keze WangAbstract
Video understanding requires not only recognizing visual content but also performing temporally grounded, multi-step reasoning over long and noisy observations. We propose Process-of-Thought (PoT) Reasoning for Videos, a framework that makes the reasoning process explicit by structuring video inference into a sequence of lightweight, verifiable steps. PoT interleaves (i) temporal evidence selection, (ii) step-wise state updates, and (iii) constrained answer synthesis, enabling the model to progressively refine hypotheses while maintaining traceability to video evidence. The framework is designed to be model-agnostic and can be plugged into existing vision-language backbones, supporting both closed-book reasoning and evidence-augmented reasoning with external tools. We further introduce a unified representation for PoT traces that aligns intermediate decisions with temporal segments, which improves robustness to distractors and reduces hallucinated explanations. Extensive experiments on standard video reasoning tasks demonstrate that PoT consistently improves factual correctness and temporal grounding, while providing interpretable reasoning traces for diagnosis and downstream use.
Abstract (translated)
视频理解不仅需要识别视觉内容，还需要对长时间且可能存在噪声的观察进行基于时间顺序的多步骤推理。为此，我们提出了用于视频的“过程思维”（PoT）推理框架，该框架通过将视频推断结构化为一系列轻量级、可验证的步骤来使推理过程透明化。PoT 交替执行以下操作：(i) 时间证据选择；(ii) 分步状态更新；以及 (iii) 约束性答案合成，这使得模型能够逐步细化假设并保持对视频证据的追溯能力。 该框架是模型无关的，并且可以插入现有的视觉-语言骨干网络中，支持闭卷推理和借助外部工具增强证据的推理。我们进一步引入了一种统一表示法来为 PoT 跟踪提供时间片段与中间决策之间的对齐，从而增强了对干扰物的鲁棒性并减少了虚构解释的数量。 在标准视频推理任务上的广泛实验表明，PoT 在提高事实准确性、基于时间顺序的基础以及提供可解释的推理跟踪方面始终如一地带来了改进，这些跟踪可用于诊断和下游应用。
URL
https://arxiv.org/abs/2602.07689
PDF
-
AD-MIR: Bridging the Gap from Perception to Persuasion in Advertising Video Understanding via Structured Reasoning
2026-02-07 17:14:06Binxiao Xu, Junyu Feng, Xiaopeng Lin, Haodong Li, Zhiyuan Feng, Bohan Zeng, Shaolin Lu, Ming Lu, Qi She, Wentao ZhangAbstract
Multimodal understanding of advertising videos is essential for interpreting the intricate relationship between visual storytelling and abstract persuasion strategies. However, despite excelling at general search, existing agents often struggle to bridge the cognitive gap between pixel-level perception and high-level marketing logic. To address this challenge, we introduce AD-MIR, a framework designed to decode advertising intent via a two-stage architecture. First, in the Structure-Aware Memory Construction phase, the system converts raw video into a structured database by integrating semantic retrieval with exact keyword matching. This approach prioritizes fine-grained brand details (e.g., logos, on-screen text) while dynamically filtering out irrelevant background noise to isolate key protagonists. Second, the Structured Reasoning Agent mimics a marketing expert through an iterative inquiry loop, decomposing the narrative to deduce implicit persuasion tactics. Crucially, it employs an evidence-based self-correction mechanism that rigorously validates these insights against specific video frames, automatically backtracking when visual support is lacking. Evaluation on the AdsQA benchmark demonstrates that AD-MIR achieves state-of-the-art performance, surpassing the strongest general-purpose agent, DVD, by 1.8% in strict and 9.5% in relaxed accuracy. These results underscore that effective advertising understanding demands explicitly grounding abstract marketing strategies in pixel-level evidence. The code is available at this https URL.
Abstract (translated)
广告视频的多模态理解对于解读视觉叙述与抽象说服策略之间的复杂关系至关重要。然而，尽管现有代理在通用搜索方面表现出色，但它们通常难以弥合像素级感知和高层次营销逻辑之间的认知差距。为了解决这一挑战，我们引入了AD-MIR框架，该框架通过两阶段架构来解码广告意图。 第一阶段是结构感知记忆构建（Structure-Aware Memory Construction）阶段，在这个阶段中，系统将原始视频转换成一个有组织的数据库，通过结合语义检索与精确关键词匹配的方式进行。这种方法优先考虑细粒度的品牌细节（如商标、屏幕文本），同时动态过滤掉不相关的背景噪音以隔离关键元素。 第二阶段是结构化推理代理（Structured Reasoning Agent）阶段，在这个阶段中，系统模仿营销专家的行为，通过迭代查询循环分解叙述并推断隐含的说服策略。尤为重要的是，它采用基于证据的自我校正机制，严格验证这些见解与特定视频帧的一致性，并在缺乏视觉支持时自动回溯。 在AdsQA基准测试上的评估表明，AD-MIR达到了最先进的性能，在严格的准确性上比最强的通用代理DVD高出1.8%，在宽松的标准下高出9.5%。这一结果强调了有效的广告理解需要将抽象的营销策略明确地建立在像素级证据的基础上。代码可在此URL处获取：[提供链接]
URL
https://arxiv.org/abs/2602.07625
PDF
