Abstract
In text-video retrieval, auxiliary captions are often used to enhance video understanding, bridging the gap between the modalities. While recent advances in multi-modal large language models (MLLMs) have enabled strong zero-shot caption generation, we observe that such captions tend to be generic and indistinguishable across visually similar videos, limiting their utility for fine-grained retrieval. Moreover, conventional captioning approaches are typically evaluated using language generation metrics, such as BLEU, which are not typically tailored for retrieval tasks that require making discriminative distinctions between candidates. To address this, we propose $\textbf{CaRe-DPO}$, a retrieval framework that directly optimizes caption generation using retrieval relevance scores. At its core is Dual-Group Direct Preference Optimization (DG-DPO), a novel learning strategy that supervises captioning by modeling preferences across groups of distinct video and caption pairs. In addition, we present an MLLM-based retrieval model that incorporates role-embeddings to better distinguish between textual inputs with different functional roles, such as an auxiliary caption and a text query. Through extensive experiments, we demonstrate that CaRe-DPO significantly enhances retrieval performance by effectively leveraging auxiliary knowledge to generate fine-grained captions for retrieval. Code is available at this https URL.
Abstract (translated)
在文本-视频检索中，辅助字幕常被用来增强对视频的理解，从而弥合不同模态之间的差距。尽管最近多模态大型语言模型（MLLM）的发展已经实现了强大的零样本字幕生成能力，但我们观察到这种字幕通常缺乏特异性，在视觉相似的视频之间难以区分，因此在精细检索任务中实用性有限。此外，传统的字幕生成方法主要通过诸如BLEU等语言生成指标进行评估，这些指标并不特别适合需要对候选项做出区别性判断的检索任务。为解决这一问题，我们提出了$\textbf{CaRe-DPO}$，一种新的检索框架，它直接利用检索相关分数来优化字幕生成过程。该框架的核心是双组直接偏好优化（Dual-Group Direct Preference Optimization, DG-DPO），这是一种新颖的学习策略，通过模拟不同视频和字幕对之间的偏好差异来进行监督学习。此外，我们还提出了一种基于MLLM的检索模型，该模型引入角色嵌入以更好地区分具有不同功能的角色的文本输入，例如辅助字幕和文本查询。 通过广泛的实验，我们证明了CaRe-DPO能够显著提升检索性能，有效地利用辅助信息生成用于检索任务的细粒度字幕。代码可在上述网址获取。
URL
https://arxiv.org/abs/2509.16560
