Video_Retrieval
-
OmniRet: Efficient and High-Fidelity Omni Modality Retrieval
2026-03-02 17:19:55Chuong Huynh, Manh Luong, Abhinav ShrivastavaAbstract
Multimodal retrieval is the task of aggregating information from queries across heterogeneous modalities to retrieve desired targets. State-of-the-art multimodal retrieval models can understand complex queries, yet they are typically limited to two modalities: text and vision. This limitation impedes the development of universal retrieval systems capable of comprehending queries that combine more than two modalities. To advance toward this goal, we present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio. Our OmniRet model addresses two critical challenges for universal retrieval: computational efficiency and representation fidelity. First, feeding massive token sequences from modality-specific encoders to Large Language Models (LLMs) is computationally inefficient. We therefore introduce an attention-based resampling mechanism to generate compact, fixed-size representations from these sequences. Second, compressing rich omni-modal data into a single embedding vector inevitably causes information loss and discards fine-grained details. We propose Attention Sliced Wasserstein Pooling to preserve these fine-grained details, leading to improved omni-modal representations. OmniRet is trained on an aggregation of approximately 6 million query-target pairs spanning 30 datasets. We benchmark our model on 13 retrieval tasks and a MMEBv2 subset. Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others. Furthermore, we curate a new Audio-Centric Multimodal Benchmark (ACM). This new benchmark introduces two critical, previously missing tasks-composed audio retrieval and audio-visual retrieval to more comprehensively evaluate a model's omni-modal embedding capacity.
Abstract (translated)
多模态检索的任务是从异构模式的查询中聚合信息，以检索出所需的目标。当前最先进的多模态检索模型能够理解复杂的查询，但它们通常仅限于处理两种模式：文本和视觉。这种限制阻碍了开发能够理解结合超过两种模式的查询的通用检索系统的发展。为了朝着这一目标迈进，我们提出了OmniRet——首个能够处理跨越三种关键模式（文本、视觉和音频）的复杂组合查询的检索模型。我们的OmniRet模型解决了通用检索面临的两个重要挑战：计算效率和表示保真度。 首先，将大量来自特定模式编码器的标记序列输入大型语言模型（LLMs）在计算上是低效的。因此，我们引入了一种基于注意力机制的重采样机制来从这些序列中生成紧凑且固定大小的表示形式。其次，将丰富的跨模态数据压缩成单个嵌入向量不可避免地会导致信息丢失，并丢弃细粒度的细节。为此，我们提出了注意切片瓦瑟斯坦池化（Attention Sliced Wasserstein Pooling）以保留这些细粒度的细节，从而提高跨模态表示的质量。 OmniRet是在一个包含约600万查询-目标对的数据集汇总上进行训练的，这些数据横跨30个不同的数据集。我们在13个检索任务和MMEBv2的一个子集上基准测试了我们的模型。在组合查询、音频和视频检索任务中，我们的模型显示出了显著改进，而在其他任务上的表现与当前最先进水平相当。 此外，我们还策划了一个新的以音频为中心的多模态基准（Audio-Centric Multimodal Benchmark, ACM）。这个新基准引入了两个关键且之前缺失的任务：组合音频检索和音视频检索。这两个任务旨在更全面地评估模型在跨模式嵌入能力方面的表现。
URL
https://arxiv.org/abs/2603.02098
PDF
-
Unified Vision-Language Modeling via Concept Space Alignment
2026-03-01 13:22:44Yifu Qiu, Paul-Ambroise Duquenne, Holger SchwenkAbstract
We introduce V-SONAR, a vision-language embedding space extended from the text-only embedding space SONAR (Omnilingual Embeddings Team et al., 2026), which supports 1500 text languages and 177 speech languages. To construct V-SONAR, we propose a post-hoc alignment pipeline that maps the representations of an existing vision encoder into the SONAR space. We thoroughly evaluate V-SONAR and show that its embeddings achieve competitive performance on text-to-video retrieval. Equipped with the OMNISONAR text decoder, V-SONAR further surpasses state-of-the-art vision-language models on video captioning tasks, including DREAM-1K (BLEU 23.9 vs. 19.6) and PE-VIDEO (BLEU 39.0 vs. 30.0). Leveraging V-SONAR, we first demonstrate that the Large Concept Model (LCM; LCM team et al. 2024) operating in SONAR and trained with English text only, can perform both single- and multi-visual concept understanding in a zero-shot manner. Finally, we introduce V-LCM, which extends the LCM with vision-language instruction tuning. V-LCM encodes vision and language inputs into an unified sequence of latent embeddings via V-SONAR and SONAR, and it is trained with the same latent diffusion objective for next-embedding prediction as in LCM's text-only pre-training. Experiments on a large-scale multilingual and -modal instruction-tuning data mixture highlight the potential of V-LCM: V-LCM matches state-of-the-art vision-language models on tasks covering image/video captioning and question answering, while significantly outperforming them across 61 rich- to low-resource languages out of all 62 tested languages.
Abstract (translated)
我们介绍了V-SONAR，这是一个从仅文本嵌入空间SONAR（Omnilingual Embeddings Team等，2026年）扩展而来的视觉-语言嵌入空间，支持1500种文字语言和177种语音语言。为了构建V-SONAR，我们提出了一种后处理对齐流程，该流程将现有视觉编码器的表示映射到SONAR空间中。我们全面评估了V-SONAR，并展示了其嵌入在文本到视频检索任务上取得了竞争性性能。配备了OMNISONAR文本解码器，V-SONAR进一步超越了最新的视觉-语言模型，在包括DREAM-1K（BLEU 23.9 vs. 19.6）和PE-VIDEO（BLEU 39.0 vs. 30.0）的视频字幕任务上。 利用V-SONAR，我们首先展示了在SONAR中运行并仅使用英语文本进行训练的大概念模型(LCM；LCM团队等,2024)可以零样本地执行单个和多个视觉概念理解。最后，我们介绍了V-LCM，它将视觉-语言指令微调扩展到了LCM中。V-LCM通过V-SONAR和SONAR将视觉和语言输入编码为统一的潜在嵌入序列，并且它的训练使用了与LCM仅文本预训练相同的潜扩散目标进行下一个嵌入预测。 在大规模多语种和多模态指令微调数据集上的实验突显了V-LCM的潜力：在涵盖图像/视频字幕和问答任务上，V-LCM的表现可媲美最新的视觉-语言模型；而在所有62种测试语言中，其在包含从丰富资源到低资源在内的61种语言中的表现显著超越。
URL
https://arxiv.org/abs/2603.01096
PDF
-
VQPP: Video Query Performance Prediction Benchmark
2026-02-19 20:32:25Adrian Catalin Lutu, Eduard Poesina, Radu Tudor IonescuAbstract
Query performance prediction (QPP) is an important and actively studied information retrieval task, having various applications, such as query reformulation, query expansion, and retrieval system selection, among many others. The task has been primarily studied in the context of text and image retrieval, whereas QPP for content-based video retrieval (CBVR) remains largely underexplored. To this end, we propose the first benchmark for video query performance prediction (VQPP), comprising two text-to-video retrieval datasets and two CBVR systems, respectively. VQPP contains a total of 56K text queries and 51K videos, and comes with official training, validation and test splits, fostering direct comparisons and reproducible results. We explore multiple pre-retrieval and post-retrieval performance predictors, creating a representative benchmark for future exploration of QPP in the video domain. Our results show that pre-retrieval predictors obtain competitive performance, enabling applications before performing the retrieval step. We also demonstrate the applicability of VQPP by employing the best performing pre-retrieval predictor as reward model for training a large language model (LLM) on the query reformulation task via direct preference optimization (DPO). We release our benchmark and code at this https URL.
Abstract (translated)
查询性能预测（QPP）是一项重要的信息检索任务，近年来受到了广泛的研究和关注。它在诸如查询重构、查询扩展以及检索系统选择等众多领域有着各种各样的应用。然而，迄今为止，大部分关于QPP的研究主要集中在文本和图像的检索上，而针对基于内容的视频检索（CBVR）中的查询性能预测问题，则几乎未被深入探索。 为此，我们提出了首个用于视频查询性能预测（VQPP）的基准测试集，该测试包括两个从文本到视频的检索数据集以及两种不同的CBVR系统。我们的VQPP包含总计56,000个文本查询和51,000段视频，并且提供官方的训练、验证及测试分组，以促进直接对比并保证研究结果可重复性。我们探索了多种预检索和后检索性能预测器，为未来在视频领域内QPP的研究提供了具有代表性的基准参考。 我们的结果显示，预检索预测器可以取得相当好的表现，这使得它们能够应用于实际的检索步骤之前的应用场景中。此外，通过使用最佳性能的预检索预测器作为直接偏好优化（DPO）奖励模型来训练大语言模型（LLM），在查询重构任务上进行学习，我们展示了VQPP的实际应用潜力。 我们的基准测试和代码已经发布在这个链接：[此URL]（请将方括号中的文本替换为实际发布的网址）。
URL
https://arxiv.org/abs/2602.17814
PDF
-
DataCube: A Video Retrieval Platform via Natural Language Semantic Profiling
2026-02-18 07:12:01Yiming Ju, Hanyu Zhao, Quanyue Ma, Donglin Hao, Chengwei Wu, Ming Li, Songjing Wang, Tengfei PanAbstract
Large-scale video repositories are increasingly available for modern video understanding and generation tasks. However, transforming raw videos into high-quality, task-specific datasets remains costly and inefficient. We present DataCube, an intelligent platform for automatic video processing, multi-dimensional profiling, and query-driven retrieval. DataCube constructs structured semantic representations of video clips and supports hybrid retrieval with neural re-ranking and deep semantic matching. Through an interactive web interface, users can efficiently construct customized video subsets from massive repositories for training, analysis, and evaluation, and build searchable systems over their own private video collections. The system is publicly accessible at this https URL. Demo Video: this https URL
Abstract (translated)
URL
https://arxiv.org/abs/2602.16231
PDF
-
Beyond Closed-Pool Video Retrieval: A Benchmark and Agent Framework for Real-World Video Search and Moment Localization
2026-02-10 03:50:59Tao Yu, Yujia Yang, Haopeng Jin, Junhao Gong, Xinlong Chen, Yuxuan Zhou, Shanbin Zhang, Jiabing Yang, Xinming Wang, Hongzhu Yi, Ping Nie, Kai Zou, Zhang Zhang, Yan Huang, Liang Wang, Yeshani, Ruiwen Tao, Jin Ma, Haijin Liang, Jinwen LuoAbstract
Traditional video retrieval benchmarks focus on matching precise descriptions to closed video pools, failing to reflect real-world searches characterized by fuzzy, multi-dimensional memories on the open web. We present \textbf{RVMS-Bench}, a comprehensive system for evaluating real-world video memory search. It consists of \textbf{1,440 samples} spanning \textbf{20 diverse categories} and \textbf{four duration groups}, sourced from \textbf{real-world open-web videos}. RVMS-Bench utilizes a hierarchical description framework encompassing \textbf{Global Impression, Key Moment, Temporal Context, and Auditory Memory} to mimic realistic multi-dimensional search cues, with all samples strictly verified via a human-in-the-loop protocol. We further propose \textbf{RACLO}, an agentic framework that employs abductive reasoning to simulate the human ``Recall-Search-Verify'' cognitive process, effectively addressing the challenge of searching for videos via fuzzy memories in the real world. Experiments reveal that existing MLLMs still demonstrate insufficient capabilities in real-world Video Retrieval and Moment Localization based on fuzzy memories. We believe this work will facilitate the advancement of video retrieval robustness in real-world unstructured scenarios.
Abstract (translated)
传统的视频检索基准主要集中在将精确描述与封闭的视频库进行匹配，未能反映开放网络中由模糊、多维度记忆特征的真实世界搜索。我们提出了**RVMS-Bench**，这是一个用于评估现实世界视频记忆搜索的全面系统。该系统包含**1,440个样本**，涵盖**20种不同的类别**和**四个持续时间组别**，所有数据均来自**真实世界的开放网络视频**。RVMS-Bench利用一个分层描述框架，包括**全局印象、关键时刻、时间背景以及听觉记忆**，以模仿现实世界中多维度的搜索线索，并通过人机交互协议对所有样本进行严格验证。 我们进一步提出了**RACLO**，这是一种代理框架，它使用溯因推理来模拟人类的“回忆-搜索-验证”认知过程，有效地解决了基于模糊记忆在现实环境中寻找视频这一挑战。实验结果显示，现有的多模态语言模型（MLLMs）在基于模糊记忆的真实世界视频检索和时刻定位方面仍表现出不足的能力。 我们认为这项工作将有助于推动在实际无结构场景中视频检索鲁棒性的进步。
URL
https://arxiv.org/abs/2602.10159
PDF
-
VidVec: Unlocking Video MLLM Embeddings for Video-Text Retrieval
2026-02-08 19:39:32Issar Tzachor, Dvir Samuel, Rami Ben-AriAbstract
Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for vision tasks, typically through fine-tuning to produce universal representations. However, their performance on video remains inferior to Video Foundation Models (VFMs). In this paper, we focus on leveraging MLLMs for video-text embedding and retrieval. We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information. Leveraging this insight, we demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training. Building on these findings, we introduce a lightweight text-based alignment strategy which maps dense video captions to short summaries and enables task-related video-text embedding learning without visual supervision. Remarkably, without any fine-tuning beyond text, our method outperforms current methods, often by a substantial margin, achieving state-of-the-art results across common video retrieval benchmarks.
Abstract (translated)
近期的研究将生成型多模态大规模语言模型（MLLM）适应为视觉任务的嵌入抽取器，通常通过微调来产生通用表示。然而，在视频处理方面，它们的表现仍不及视频基础模型（VFMs）。本文专注于利用MLLM进行视频-文本嵌入和检索。 首先，我们进行了系统的逐层分析，显示中间层的预训练MLLM已经编码了大量与任务相关的信息。基于这一发现，我们展示了将中间层嵌入与校准后的MLLM头相结合，在无需任何训练的情况下就能实现强大的零样本检索性能。 在此基础上，我们提出了一种轻量级的文本对齐策略，该策略可以将密集视频描述映射到简短摘要，并在没有视觉监督的情况下学习任务相关的视频-文本嵌入。值得注意的是，除了文本外不进行任何微调，我们的方法超越了现有方法，在多个常见的视频检索基准测试中取得了最先进的结果。
URL
https://arxiv.org/abs/2602.08099
PDF
-
Human-in-the-loop Adaptation in Group Activity Feature Learning for Team Sports Video Retrieval
2026-02-03 06:15:43Chihiro Nakatani, Hiroaki Kawashima, Norimichi UkitaAbstract
This paper proposes human-in-the-loop adaptation for Group Activity Feature Learning (GAFL) without group activity annotations. This human-in-the-loop adaptation is employed in a group-activity video retrieval framework to improve its retrieval performance. Our method initially pre-trains the GAF space based on the similarity of group activities in a self-supervised manner, unlike prior work that classifies videos into pre-defined group activity classes in a supervised learning manner. Our interactive fine-tuning process updates the GAF space to allow a user to better retrieve videos similar to query videos given by the user. In this fine-tuning, our proposed data-efficient video selection process provides several videos, which are selected from a video database, to the user in order to manually label these videos as positive or negative. These labeled videos are used to update (i.e., fine-tune) the GAF space, so that the positive and negative videos move closer to and farther away from the query videos through contrastive learning. Our comprehensive experimental results on two team sports datasets validate that our method significantly improves the retrieval performance. Ablation studies also demonstrate that several components in our human-in-the-loop adaptation contribute to the improvement of the retrieval performance. Code: this https URL.
Abstract (translated)
本文提出了一种无需群体活动标注的人机交互适应方法，用于群体活动特征学习（GAFL）。该人机交互适应技术被应用到一个群体活动视频检索框架中，以提升其检索性能。我们的方法首先通过自监督方式基于群体活动的相似性来预训练GAF空间，与以往的研究不同的是，后者在有监督的学习模式下将视频分类为预先定义好的群体活动类别。 我们提出的方法包括一种交互式的微调过程，该过程更新GAF空间以使用户能够更好地检索到与其查询视频类似的视频。在此微调过程中，我们的高效数据选择流程从一个视频数据库中选取几个视频，并提供给用户手动标注这些视频是正样本还是负样本。之后，根据这些标签更新（即细化）GAF空间，从而通过对比学习方法让正样本和负样本与查询视频的距离更近或更远。 我们在两个团队运动数据集上的全面实验结果验证了我们的方法显著提升了检索性能。消融研究表明我们的人机交互适应中的几个组件对提高检索性能有贡献作用。代码链接：[请将URL替换为实际提供的链接]。
URL
https://arxiv.org/abs/2602.03157
PDF
-
RANKVIDEO: Reasoning Reranking for Text-to-Video Retrieval
2026-02-02 18:40:37Tyler Skow, Alexander Martin, Benjamin Van Durme, Rama Chellappa, Reno KrizAbstract
Reranking is a critical component of modern retrieval systems, which typically pair an efficient first-stage retriever with a more expressive model to refine results. While large reasoning models have driven rapid progress in text-centric reranking, reasoning-based reranking for video retrieval remains underexplored. To address this gap, we introduce RANKVIDEO, a reasoning-based reranker for video retrieval that explicitly reasons over query-video pairs using video content to assess relevance. RANKVIDEO is trained using a two-stage curriculum consisting of perception-grounded supervised fine-tuning followed by reranking training that combines pointwise, pairwise, and teacher confidence distillation objectives, and is supported by a data synthesis pipeline for constructing reasoning-intensive query-video pairs. Experiments on the large-scale MultiVENT 2.0 benchmark demonstrate that RANKVIDEO consistently improves retrieval performance within a two-stage framework, yielding an average improvement of 31% on nDCG@10 and outperforming text-only and vision-language reranking alternatives, while more efficient.
Abstract (translated)
URL
https://arxiv.org/abs/2602.02444
PDF
-
ShotFinder: Imagination-Driven Open-Domain Video Shot Retrieval via Web Search
2026-01-30 18:01:17Tao Yu, Haopeng Jin, Hao Wang, Shenghua Chai, Yujia Yang, Junhao Gong, Jiaming Guo, Minghui Zhang, Xinlong Chen, Zhenghao Zhang, Yuxuan Zhou, Yanpei Gong, YuanCheng Liu, Yiming Ding, Kangwei Zeng, Pengfei Yang, Zhongtian Luo, Yufei Xiong, Shanbin Zhang, Shaoxiong Cheng, Huang Ruilin, Li Shuo, Yuxi Niu, Xinyuan Zhang, Yueya Xu, Jie Mao, Ruixuan Ji, Yaru Zhao, Mingchen Zhang, Jiabing Yang, Jiaqi Liu, YiFan Zhang, Hongzhu Yi, Xinming Wang, Cheng Zhong, Xiao Ma, Zhang Zhang, Yan Huang, Liang WangAbstract
In recent years, large language models (LLMs) have made rapid progress in information retrieval, yet existing research has mainly focused on text or static multimodal settings. Open-domain video shot retrieval, which involves richer temporal structure and more complex semantics, still lacks systematic benchmarks and analysis. To fill this gap, we introduce ShotFinder, a benchmark that formalizes editing requirements as keyframe-oriented shot descriptions and introduces five types of controllable single-factor constraints: Temporal order, Color, Visual style, Audio, and Resolution. We curate 1,210 high-quality samples from YouTube across 20 thematic categories, using large models for generation with human verification. Based on the benchmark, we propose ShotFinder, a text-driven three-stage retrieval and localization pipeline: (1) query expansion via video imagination, (2) candidate video retrieval with a search engine, and (3) description-guided temporal localization. Experiments on multiple closed-source and open-source models reveal a significant gap to human performance, with clear imbalance across constraints: temporal localization is relatively tractable, while color and visual style remain major challenges. These results reveal that open-domain video shot retrieval is still a critical capability that multimodal large models have yet to overcome.
Abstract (translated)
URL
https://arxiv.org/abs/2601.23232
PDF
-
CoVA: Text-Guided Composed Video Retrieval for Audio-Visual Content
Abstract
Composed Video Retrieval (CoVR) aims to retrieve a target video from a large gallery using a reference video and a textual query specifying visual modifications. However, existing benchmarks consider only visual changes, ignoring videos that differ in audio despite visual similarity. To address this limitation, we introduce Composed retrieval for Video with its Audio CoVA, a new retrieval task that accounts for both visual and auditory variations. To support this, we construct AV-Comp, a benchmark consisting of video pairs with cross-modal changes and corresponding textual queries that describe the differences. We also propose AVT Compositional Fusion (AVT), which integrates video, audio, and text features by selectively aligning the query to the most relevant modality. AVT outperforms traditional unimodal fusion and serves as a strong baseline for CoVA. Examples from the proposed dataset, including both visual and auditory information, are available at this https URL.
Abstract (translated)
URL
https://arxiv.org/abs/2601.22508
PDF
-
Generative Recall, Dense Reranking: Learning Multi-View Semantic IDs for Efficient Text-to-Video Retrieval
2026-01-29 02:49:33Zecheng Zhao, Zhi Chen, Zi Huang, Shazia Sadiq, Tong ChenAbstract
Text-to-Video Retrieval (TVR) is essential in video platforms. Dense retrieval with dual-modality encoders leads in accuracy, but its computation and storage scale poorly with corpus size. Thus, real-time large-scale applications adopt two-stage retrieval, where a fast recall model gathers a small candidate pool, which is reranked by an advanced dense retriever. Due to hugely reduced candidates, the reranking model can use any off-the-shelf dense retriever without hurting efficiency, meaning the recall model bounds two-stage TVR performance. Recently, generative retrieval (GR) replaces dense video embeddings with discrete semantic IDs and retrieves by decoding text queries into ID tokens. GR offers near-constant inference and storage complexity, and its semantic IDs capture high-level video features via quantization, making it ideal for quickly eliminating irrelevant candidates during recall. However, as a recall model in two-stage TVR, GR suffers from (i) semantic ambiguity, where each video satisfies diverse queries but is forced into one semantic ID; and (ii) cross-modal misalignment, as semantic IDs are solely derived from visual features without text supervision. We propose Generative Recall and Dense Reranking (GRDR), designing a novel GR method to uplift recalled candidate quality. GRDR assigns multiple semantic IDs to each video using a query-guided multi-view tokenizer exposing diverse semantic access paths, and jointly trains the tokenizer and generative retriever via a shared codebook to cast semantic IDs as the semantic bridge between texts and videos. At inference, trie-constrained decoding generates a compact candidate set reranked by a dense model for fine-grained matching. Experiments on TVR benchmarks show GRDR matches strong dense retrievers in accuracy while reducing index storage by an order of magnitude and accelerating up to 300$\times$ in full-corpus retrieval.
Abstract (translated)
URL
https://arxiv.org/abs/2601.21193
PDF
-
StructAlign: Structured Cross-Modal Alignment for Continual Text-to-Video Retrieval
2026-01-28 13:34:44Shaokun Wang, Weili Guan, Jizhou Han, Jianlong Wu, Yupeng Hu, Liqiang NieAbstract
Continual Text-to-Video Retrieval (CTVR) is a challenging multimodal continual learning setting, where models must incrementally learn new semantic categories while maintaining accurate text-video alignment for previously learned ones, thus making it particularly prone to catastrophic forgetting. A key challenge in CTVR is feature drift, which manifests in two forms: intra-modal feature drift caused by continual learning within each modality, and non-cooperative feature drift across modalities that leads to modality misalignment. To mitigate these issues, we propose StructAlign, a structured cross-modal alignment method for CTVR. First, StructAlign introduces a simplex Equiangular Tight Frame (ETF) geometry as a unified geometric prior to mitigate modality misalignment. Building upon this geometric prior, we design a cross-modal ETF alignment loss that aligns text and video features with category-level ETF prototypes, encouraging the learned representations to form an approximate simplex ETF geometry. In addition, to suppress intra-modal feature drift, we design a Cross-modal Relation Preserving loss, which leverages complementary modalities to preserve cross-modal similarity relations, providing stable relational supervision for feature updates. By jointly addressing non-cooperative feature drift across modalities and intra-modal feature drift, StructAlign effectively alleviates catastrophic forgetting in CTVR. Extensive experiments on benchmark datasets demonstrate that our method consistently outperforms state-of-the-art continual retrieval approaches.
Abstract (translated)
URL
https://arxiv.org/abs/2601.20597
PDF
-
X-Aligner: Composed Visual Retrieval without the Bells and Whistles
2026-01-23 09:33:38Yuqian Zheng, Mariana-Iuliana GeorgescuAbstract
Composed Video Retrieval (CoVR) facilitates video retrieval by combining visual and textual queries. However, existing CoVR frameworks typically fuse multimodal inputs in a single stage, achieving only marginal gains over initial baseline. To address this, we propose a novel CoVR framework that leverages the representational power of Vision Language Models (VLMs). Our framework incorporates a novel cross-attention module X-Aligner, composed of cross-attention layers that progressively fuse visual and textual inputs and align their multimodal representation with that of the target video. To further enhance the representation of the multimodal query, we incorporate the caption of the visual query as an additional input. The framework is trained in two stages to preserve the pretrained VLM representation. In the first stage, only the newly introduced module is trained, while in the second stage, the textual query encoder is also fine-tuned. We implement our framework on top of BLIP-family architecture, namely BLIP and BLIP-2, and train it on the Webvid-CoVR data set. In addition to in-domain evaluation on Webvid-CoVR-Test, we perform zero-shot evaluations on the Composed Image Retrieval (CIR) data sets CIRCO and Fashion-IQ. Our framework achieves state-of-the-art performance on CoVR obtaining a Recall@1 of 63.93% on Webvid-CoVR-Test, and demonstrates strong zero-shot generalization on CIR tasks.
Abstract (translated)
URL
https://arxiv.org/abs/2601.16582
PDF
-
HVD: Human Vision-Driven Video Representation Learning for Text-Video Retrieval
2026-01-22 17:57:42Zequn Xie, Xin Liu, Boyun Zhang, Yuxiao Lin, Sihang Cai, Tao JinAbstract
The success of CLIP has driven substantial progress in text-video retrieval. However, current methods often suffer from "blind" feature interaction, where the model struggles to discern key visual information from background noise due to the sparsity of textual queries. To bridge this gap, we draw inspiration from human cognitive behavior and propose the Human Vision-Driven (HVD) model. Our framework establishes a coarse-to-fine alignment mechanism comprising two key components: the Frame Features Selection Module (FFSM) and the Patch Features Compression Module (PFCM). FFSM mimics the human macro-perception ability by selecting key frames to eliminate temporal redundancy. Subsequently, PFCM simulates micro-perception by aggregating patch features into salient visual entities through an advanced attention mechanism, enabling precise entity-level matching. Extensive experiments on five benchmarks demonstrate that HVD not only captures human-like visual focus but also achieves state-of-the-art performance.
Abstract (translated)
CLIP模型的成功推动了文本视频检索领域的显著进步。然而，当前的方法往往在“盲”特征交互方面存在问题，即由于文本查询的稀疏性，模型难以从背景噪声中区分出关键视觉信息。为了弥补这一差距，我们借鉴了人类的认知行为，并提出了人眼视图驱动（HVD）模型。我们的框架建立了一个由粗到细的对齐机制，包含两个关键组件：帧特征选择模块（FFSM）和补丁特征压缩模块（PFCM）。FFSM通过选择关键帧来消除时间冗余，模拟了人类宏观感知能力。随后，PFCM通过先进的注意力机制聚合补丁特征以形成显著视觉实体，从而模仿微观感知并实现精确的实体级别匹配。 在五个基准测试中的大量实验表明，HVD不仅能够捕捉到类似人的视觉关注点，还实现了最先进的性能。
URL
https://arxiv.org/abs/2601.16155
PDF
-
PREGEN: Uncovering Latent Thoughts in Composed Video Retrieval
2026-01-20 09:57:04Gabriele Serussi, David Vainshtein, Jonathan Kouchly, Dotan Di Castro, Chaim BaskinAbstract
Composed Video Retrieval (CoVR) aims to retrieve a video based on a query video and a modifying text. Current CoVR methods fail to fully exploit modern Vision-Language Models (VLMs), either using outdated architectures or requiring computationally expensive fine-tuning and slow caption generation. We introduce PREGEN (PRE GENeration extraction), an efficient and powerful CoVR framework that overcomes these limitations. Our approach uniquely pairs a frozen, pre-trained VLM with a lightweight encoding model, eliminating the need for any VLM fine-tuning. We feed the query video and modifying text into the VLM and extract the hidden state of the final token from each layer. A simple encoder is then trained on these pooled representations, creating a semantically rich and compact embedding for retrieval. PREGEN significantly advances the state of the art, surpassing all prior methods on standard CoVR benchmarks with substantial gains in Recall@1 of +27.23 and +69.59. Our method demonstrates robustness across different VLM backbones and exhibits strong zero-shot generalization to more complex textual modifications, highlighting its effectiveness and semantic capabilities.
Abstract (translated)
组合视频检索（CoVR）的目标是基于查询视频和修改文本来检索相关视频。当前的CoVR方法未能充分利用现代视觉-语言模型（VLM），要么使用过时的架构，要么需要昂贵的计算资源进行微调，并且会产生缓慢的字幕生成过程。我们引入了PREGEN（PRE GENeration extraction），这是一种高效的组合视频检索框架，能够克服这些限制。我们的方法独特地将一个冻结、预训练的VLM与一个轻量级编码模型配对，从而完全消除了对任何VLM微调的需求。我们将查询视频和修改文本输入到VLM中，并从每一层提取最终令牌的隐藏状态。然后在这些汇集表示上训练一个简单的编码器，创建用于检索的语义丰富且紧凑的嵌入。PREGEN显著地推进了当前的技术水平，在标准CoVR基准测试上超越所有先前的方法，Recall@1分别提高了27.23和69.59个百分点。我们的方法在不同的VLM骨干网络上表现出强大的鲁棒性，并展示了对更复杂文本修改的零样本泛化能力，突显了其有效性和语义处理能力。
URL
https://arxiv.org/abs/2601.13797
PDF
-
VIRTUE: Versatile Video Retrieval Through Unified Embeddings
2026-01-17 23:13:38Shaunak Halbe, Bhagyashree Puranik, Jayakrishnan Unnikrishnan, Kushan Thakkar, Vimal Bhat, Toufiq ParagAbstract
Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval and fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieval performance by training modality-specific encoders on massive datasets, but they lack the ability to process composed multimodal queries. In contrast, multimodal LLM (MLLM)-based methods support rich multimodal search but their retrieval performance remains well below that of specialized systems. We present VIRTUE, an MLLM-based versatile video retrieval framework that integrates corpus and moment-level retrieval capabilities while accommodating composed multimodal queries within a single architecture. We use contrastive alignment of visual and textual embeddings generated using a shared MLLM backbone to facilitate efficient embedding-based candidate search. Our embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment retrieval, and state of the art results for zero-shot composed video retrieval. With additional training for reranking candidates identified in the embedding-based search, our model substantially outperforms existing MLLM-based retrieval systems and achieves retrieval performance comparable to state of the art specialized models which are trained on orders of magnitude larger data.
Abstract (translated)
现代视频检索系统被期望能够处理从语料库级别的检索和细粒度时刻定位到灵活的多模态查询等多样化任务。专门架构通过在大规模数据集上训练特定模式的编码器来实现强大的检索性能，但它们缺乏处理复杂组合多模态查询的能力。相比之下，基于多模态大型语言模型（MLLM）的方法支持丰富的多模态搜索，但其检索性能仍远低于专业系统。 我们提出了VIRTUE，这是一种基于MLLM的多功能视频检索框架，在单一架构中集成了语料库和时刻级别的检索能力，并且能够处理组合多模态查询。我们使用共享MLLM骨干网生成视觉和文本嵌入并进行对比对齐，以促进高效的基于嵌入的候选搜索。 我们的嵌入模型在70万对视觉-文本数据样本上通过低秩适应（LoRA）高效训练后，在零样本视频检索任务中超越了其他所有基于MLLM的方法。此外，我们还展示了该模型可以在不进行进一步训练的情况下调整以实现与最佳水平相当的零样本时刻检索结果，并且在零样本组合视频检索方面也达到了最先进的表现。 通过为嵌入搜索中确定的候选对象进行额外重新排序培训，我们的模型显著超越了现有的基于MLLM的检索系统，并实现了与专用于大规模数据的专业模型相媲美的检索性能。
URL
https://arxiv.org/abs/2601.12193
PDF
-
SMc2f: Robust Scenario Mining for Robotic Autonomy from Coarse to Fine
2026-01-17 11:25:55Yifei Chen, Ross GreerAbstract
The safety validation of autonomous robotic vehicles hinges on systematically testing their planning and control stacks against rare, safety-critical scenarios. Mining these long-tail events from massive real-world driving logs is therefore a critical step in the robotic development lifecycle. The goal of the Scenario Mining task is to retrieve useful information to enable targeted re-simulation, regression testing, and failure analysis of the robot's decision-making algorithms. RefAV, introduced by the Argoverse team, is an end-to-end framework that uses large language models (LLMs) to spatially and temporally localize scenarios described in natural language. However, this process performs retrieval on trajectory labels, ignoring the direct connection between natural language and raw RGB images, which runs counter to the intuition of video retrieval; it also depends on the quality of upstream 3D object detection and tracking. Further, inaccuracies in trajectory data lead to inaccuracies in downstream spatial and temporal localization. To address these issues, we propose Robust Scenario Mining for Robotic Autonomy from Coarse to Fine (SMc2f), a coarse-to-fine pipeline that employs vision-language models (VLMs) for coarse image-text filtering, builds a database of successful mining cases on top of RefAV and automatically retrieves exemplars to few-shot condition the LLM for more robust retrieval, and introduces text-trajectory contrastive learning to pull matched pairs together and push mismatched pairs apart in a shared embedding space, yielding a fine-grained matcher that refines the LLM's candidate trajectories. Experiments on public datasets demonstrate substantial gains in both retrieval quality and efficiency.
Abstract (translated)
自主机器人车辆的安全验证依赖于对其规划和控制系统进行系统性测试，尤其是在罕见的、关键安全场景下。因此，从大规模的真实世界驾驶日志中挖掘这些长尾事件是机器人开发生命周期中的一个关键步骤。情景挖掘任务的目标是从自然语言描述的情景中检索有用的信息，以支持针对特定目标的重新模拟、回归测试和故障分析。 RefAV是由Argoverse团队提出的一种端到端框架，该框架使用大型语言模型（LLMs）来空间和时间定位用自然语言描述的情景。然而，这一过程基于轨迹标签进行检索，忽略了自然语言与原始RGB图像之间的直接联系，这违背了视频检索的直觉；此外，它还依赖于上游3D目标检测和跟踪的质量。不准确的轨迹数据会导致下游的空间和时间定位不准确。 为了解决这些问题，我们提出了从粗到细的情景挖掘方法（SMc2f），这是一种采用视觉-语言模型(VLMs)进行图像文本过滤、在RefAV基础上构建成功挖掘案例数据库并自动检索实例以对LLM进行少样本条件化从而提高检索鲁棒性，并引入了基于文本和轨迹的对比学习，将匹配对拉近并在共享嵌入空间中将不匹配对推开的方法。这种方法最终形成了一种细粒度匹配器，用于精炼LLM候选轨迹。 在公共数据集上的实验表明，在检索质量和效率方面取得了显著提升。
URL
https://arxiv.org/abs/2601.12010
PDF
-
Plenoptic Video Generation
2026-01-08 18:58:32Xiao Fu, Shitao Tang, Min Shi, Xian Liu, Jinwei Gu, Ming-Yu Liu, Dahua Lin, Chen-Hsuan LinAbstract
Camera-controlled generative video re-rendering methods, such as ReCamMaster, have achieved remarkable progress. However, despite their success in single-view setting, these works often struggle to maintain consistency across multi-view scenarios. Ensuring spatio-temporal coherence in hallucinated regions remains challenging due to the inherent stochasticity of generative models. To address it, we introduce PlenopticDreamer, a framework that synchronizes generative hallucinations to maintain spatio-temporal memory. The core idea is to train a multi-in-single-out video-conditioned model in an autoregressive manner, aided by a camera-guided video retrieval strategy that adaptively selects salient videos from previous generations as conditional inputs. In addition, Our training incorporates progressive context-scaling to improve convergence, self-conditioning to enhance robustness against long-range visual degradation caused by error accumulation, and a long-video conditioning mechanism to support extended video generation. Extensive experiments on the Basic and Agibot benchmarks demonstrate that PlenopticDreamer achieves state-of-the-art video re-rendering, delivering superior view synchronization, high-fidelity visuals, accurate camera control, and diverse view transformations (e.g., third-person to third-person, and head-view to gripper-view in robotic manipulation). Project page: this https URL
Abstract (translated)
相机控制的生成视频重渲染方法，如ReCamMaster，已经取得了显著的进步。然而，在单一视角设置下尽管这些方法表现出色，但在多视角场景中却往往难以保持一致性。由于生成模型固有的随机性，确保幻觉区域的空间和时间连贯性仍然是一项挑战。为了解决这个问题，我们提出了PlenopticDreamer框架，该框架通过同步生成的幻象来维持空间和时间记忆的一致性。核心思想是训练一个以多输入单输出视频条件模型自回归方式进行训练，并辅以相机引导的视频检索策略，这种策略可以动态地从先前的生成中选择显著视频作为条件输入。此外，在训练过程中还加入了渐进式上下文扩展来提高收敛速度，自我调节来增强对长时间视觉退化（由错误累积引起）的鲁棒性，以及支持长视频生成的长期视频条件机制。在Basic和Agibot基准测试上的广泛实验表明，PlenopticDreamer实现了最先进的视频重渲染性能，在视角同步、高保真视觉效果、准确的相机控制以及多样化的视图转换（如第三人称到第三人称，或机器人操作中的头部视图到机械臂视图）等方面表现出色。项目页面：[此链接](https://this-url.com/)
URL
https://arxiv.org/abs/2601.05239
PDF
-
SOVABench: A Vehicle Surveillance Action Retrieval Benchmark for Multimodal Large Language Models
2026-01-08 10:58:59Oriol Rabasseda, Zenjie Li, Kamal Nasrollahi, Sergio EscaleraAbstract
Automatic identification of events and recurrent behavior analysis are critical for video surveillance. However, most existing content-based video retrieval benchmarks focus on scene-level similarity and do not evaluate the action discrimination required in surveillance. To address this gap, we introduce SOVABench (Surveillance Opposite Vehicle Actions Benchmark), a real-world retrieval benchmark built from surveillance footage and centered on vehicle-related actions. SOVABench defines two evaluation protocols (inter-pair and intra-pair) to assess cross-action discrimination and temporal direction understanding. Although action distinctions are generally intuitive for human observers, our experiments show that they remain challenging for state-of-the-art vision and multimodal models. Leveraging the visual reasoning and instruction-following capabilities of Multimodal Large Language Models (MLLMs), we present a training-free framework for producing interpretable embeddings from MLLM-generated descriptions for both images and videos. The framework achieves strong performance on SOVABench as well as on several spatial and counting benchmarks where contrastive Vision-Language Models often fail. The code, annotations, and instructions to construct the benchmark are publicly available.
Abstract (translated)
自动识别事件和重复行为分析对于视频监控至关重要。然而，大多数现有的基于内容的视频检索基准测试主要关注场景级别的相似性，并且不评估在监控中所需的动作辨别能力。为了弥补这一不足，我们引入了SOVABench（监控相对车辆动作基准），这是一个从监控录像构建的真实世界检索基准，专注于与车辆相关的行动。SOVABench定义了两种评估协议（跨对和内对）来评价跨动作的区分能力和时间方向的理解力。虽然动作之间的区别对于人类观察者来说通常是直观的，但我们的实验表明，这些任务仍然为最先进的视觉和多模态模型带来了挑战。 通过利用多模态大型语言模型（MLLMs）在视觉推理和指令遵循方面的能力，我们提出了一种无需训练框架，用于从MLLM生成的描述中生产出具有解释性的图像和视频嵌入。该框架在SOVABench上实现了强大的性能，并且还在多个空间和计数基准测试中表现出色，这些地方对比度视觉语言模型常常失败。 该项目的代码、标注以及构建基准的说明已公开提供。
URL
https://arxiv.org/abs/2601.04824
PDF
-
TV-RAG: A Temporal-aware and Semantic Entropy-Weighted Framework for Long Video Retrieval and Understanding
2025-12-29 14:10:22Zongsheng Cao, Yangfan He, Anran Liu, Feng Chen, Zepeng Wang, Jun XieAbstract
Large Video Language Models (LVLMs) have rapidly emerged as the focus of multimedia AI research. Nonetheless, when confronted with lengthy videos, these models struggle: their temporal windows are narrow, and they fail to notice fine-grained semantic shifts that unfold over extended durations. Moreover, mainstream text-based retrieval pipelines, which rely chiefly on surface-level lexical overlap, ignore the rich temporal interdependence among visual, audio, and subtitle channels. To mitigate these limitations, we propose TV-RAG, a training-free architecture that couples temporal alignment with entropy-guided semantics to improve long-video reasoning. The framework contributes two main mechanisms: \emph{(i)} a time-decay retrieval module that injects explicit temporal offsets into the similarity computation, thereby ranking text queries according to their true multimedia context; and \emph{(ii)} an entropy-weighted key-frame sampler that selects evenly spaced, information-dense frames, reducing redundancy while preserving representativeness. By weaving these temporal and semantic signals together, TV-RAG realises a dual-level reasoning routine that can be grafted onto any LVLM without re-training or fine-tuning. The resulting system offers a lightweight, budget-friendly upgrade path and consistently surpasses most leading baselines across established long-video benchmarks such as Video-MME, MLVU, and LongVideoBench, confirming the effectiveness of our model. The code can be found at this https URL.
Abstract (translated)
大型视频语言模型（LVLMs）作为多媒体AI研究的焦点迅速崛起。然而，面对长视频时，这些模型遇到了困难：它们的时间窗口狭窄，无法捕捉到长时间内细微的语义变化。此外，主流基于文本的检索流程主要依赖于表面级别的词汇重叠，忽视了视觉、音频和字幕通道之间的丰富时间相互依存关系。为了缓解这些问题，我们提出了TV-RAG架构，这是一种无需训练的方法，结合了时序对齐与熵引导语义来提升长视频推理能力。该框架贡献了两种主要机制：（i）一种时间衰减检索模块，在相似度计算中注入明确的时间偏移量，从而根据真实的多媒体背景排列文本查询；（ii）一种基于熵加权的关键帧采样器，选择均匀间隔且信息密集的帧，减少了冗余同时保持代表性。通过将这些时序和语义信号编织在一起，TV-RAG实现了双层推理流程，可以在不重新训练或微调的情况下与任何LVLM集成。由此产生的系统提供了一种轻量级、预算友好的升级途径，并在Video-MME、MLVU 和 LongVideoBench 等公认的长视频基准测试中持续超越大多数领先基线，证明了我们模型的有效性。该代码可在[此链接](https://this-is-the-url-you-need-to-replace.com)找到。
URL
https://arxiv.org/abs/2512.23483
PDF
