Abstract
Composed Video Retrieval (CoVR) aims to retrieve a video based on a query video and a modifying text. Current CoVR methods fail to fully exploit modern Vision-Language Models (VLMs), either using outdated architectures or requiring computationally expensive fine-tuning and slow caption generation. We introduce PREGEN (PRE GENeration extraction), an efficient and powerful CoVR framework that overcomes these limitations. Our approach uniquely pairs a frozen, pre-trained VLM with a lightweight encoding model, eliminating the need for any VLM fine-tuning. We feed the query video and modifying text into the VLM and extract the hidden state of the final token from each layer. A simple encoder is then trained on these pooled representations, creating a semantically rich and compact embedding for retrieval. PREGEN significantly advances the state of the art, surpassing all prior methods on standard CoVR benchmarks with substantial gains in Recall@1 of +27.23 and +69.59. Our method demonstrates robustness across different VLM backbones and exhibits strong zero-shot generalization to more complex textual modifications, highlighting its effectiveness and semantic capabilities.
Abstract (translated)
组合视频检索（CoVR）的目标是基于查询视频和修改文本来检索相关视频。当前的CoVR方法未能充分利用现代视觉-语言模型（VLM），要么使用过时的架构，要么需要昂贵的计算资源进行微调，并且会产生缓慢的字幕生成过程。我们引入了PREGEN（PRE GENeration extraction），这是一种高效的组合视频检索框架，能够克服这些限制。我们的方法独特地将一个冻结、预训练的VLM与一个轻量级编码模型配对，从而完全消除了对任何VLM微调的需求。我们将查询视频和修改文本输入到VLM中，并从每一层提取最终令牌的隐藏状态。然后在这些汇集表示上训练一个简单的编码器，创建用于检索的语义丰富且紧凑的嵌入。PREGEN显著地推进了当前的技术水平，在标准CoVR基准测试上超越所有先前的方法，Recall@1分别提高了27.23和69.59个百分点。我们的方法在不同的VLM骨干网络上表现出强大的鲁棒性，并展示了对更复杂文本修改的零样本泛化能力，突显了其有效性和语义处理能力。
URL
https://arxiv.org/abs/2601.13797
