Abstract
Modern video retrieval systems are expected to handle diverse tasks ranging from corpus-level retrieval and fine-grained moment localization to flexible multimodal querying. Specialized architectures achieve strong retrieval performance by training modality-specific encoders on massive datasets, but they lack the ability to process composed multimodal queries. In contrast, multimodal LLM (MLLM)-based methods support rich multimodal search but their retrieval performance remains well below that of specialized systems. We present VIRTUE, an MLLM-based versatile video retrieval framework that integrates corpus and moment-level retrieval capabilities while accommodating composed multimodal queries within a single architecture. We use contrastive alignment of visual and textual embeddings generated using a shared MLLM backbone to facilitate efficient embedding-based candidate search. Our embedding model, trained efficiently using low-rank adaptation (LoRA) on 700K paired visual-text data samples, surpasses other MLLM-based methods on zero-shot video retrieval tasks. Additionally, we demonstrate that the same model can be adapted without further training to achieve competitive results on zero-shot moment retrieval, and state of the art results for zero-shot composed video retrieval. With additional training for reranking candidates identified in the embedding-based search, our model substantially outperforms existing MLLM-based retrieval systems and achieves retrieval performance comparable to state of the art specialized models which are trained on orders of magnitude larger data.
Abstract (translated)
现代视频检索系统被期望能够处理从语料库级别的检索和细粒度时刻定位到灵活的多模态查询等多样化任务。专门架构通过在大规模数据集上训练特定模式的编码器来实现强大的检索性能，但它们缺乏处理复杂组合多模态查询的能力。相比之下，基于多模态大型语言模型（MLLM）的方法支持丰富的多模态搜索，但其检索性能仍远低于专业系统。 我们提出了VIRTUE，这是一种基于MLLM的多功能视频检索框架，在单一架构中集成了语料库和时刻级别的检索能力，并且能够处理组合多模态查询。我们使用共享MLLM骨干网生成视觉和文本嵌入并进行对比对齐，以促进高效的基于嵌入的候选搜索。 我们的嵌入模型在70万对视觉-文本数据样本上通过低秩适应（LoRA）高效训练后，在零样本视频检索任务中超越了其他所有基于MLLM的方法。此外，我们还展示了该模型可以在不进行进一步训练的情况下调整以实现与最佳水平相当的零样本时刻检索结果，并且在零样本组合视频检索方面也达到了最先进的表现。 通过为嵌入搜索中确定的候选对象进行额外重新排序培训，我们的模型显著超越了现有的基于MLLM的检索系统，并实现了与专用于大规模数据的专业模型相媲美的检索性能。
URL
https://arxiv.org/abs/2601.12193
