Abstract
Recent studies have adapted generative Multimodal Large Language Models (MLLMs) into embedding extractors for vision tasks, typically through fine-tuning to produce universal representations. However, their performance on video remains inferior to Video Foundation Models (VFMs). In this paper, we focus on leveraging MLLMs for video-text embedding and retrieval. We first conduct a systematic layer-wise analysis, showing that intermediate (pre-trained) MLLM layers already encode substantial task-relevant information. Leveraging this insight, we demonstrate that combining intermediate-layer embeddings with a calibrated MLLM head yields strong zero-shot retrieval performance without any training. Building on these findings, we introduce a lightweight text-based alignment strategy which maps dense video captions to short summaries and enables task-related video-text embedding learning without visual supervision. Remarkably, without any fine-tuning beyond text, our method outperforms current methods, often by a substantial margin, achieving state-of-the-art results across common video retrieval benchmarks.
Abstract (translated)
近期的研究将生成型多模态大规模语言模型（MLLM）适应为视觉任务的嵌入抽取器，通常通过微调来产生通用表示。然而，在视频处理方面，它们的表现仍不及视频基础模型（VFMs）。本文专注于利用MLLM进行视频-文本嵌入和检索。 首先，我们进行了系统的逐层分析，显示中间层的预训练MLLM已经编码了大量与任务相关的信息。基于这一发现，我们展示了将中间层嵌入与校准后的MLLM头相结合，在无需任何训练的情况下就能实现强大的零样本检索性能。 在此基础上，我们提出了一种轻量级的文本对齐策略，该策略可以将密集视频描述映射到简短摘要，并在没有视觉监督的情况下学习任务相关的视频-文本嵌入。值得注意的是，除了文本外不进行任何微调，我们的方法超越了现有方法，在多个常见的视频检索基准测试中取得了最先进的结果。
URL
https://arxiv.org/abs/2602.08099
