Abstract
Universal multimodal retrieval (UMR), which aims to address complex retrieval tasks where both queries and candidates span diverse modalities, has been significantly advanced by the emergence of MLLMs. While state-of-the-art MLLM-based methods in the literature predominantly adopt contrastive learning principles, they often differ in their specific training recipes. Despite their success, the mechanisms underlying their retrieval capabilities remain largely unexplored, potentially resulting in suboptimal performance and limited generalization ability. To address these issues, we present a comprehensive study aimed at uncovering the key factors that drive effective embedding learning for UMR using MLLMs. We begin by implementing a general MLLM-based embedding learning pipeline, and systematically analyze the primary contributors to high-performing universal retrieval systems. Based on this, we explore various aspects of the details in embedding generation and training strategies, including progressive transition, hard negative mining and re-ranker distillation. Notably, our findings reveal that often-overlooked factors can have a substantial impact on model performance. Building on these discoveries, we introduce a unified framework termed U-MARVEL (\textbf{U}niversal \textbf{M}ultimod\textbf{A}l \textbf{R}etrie\textbf{V}al via \textbf{E}mbedding \textbf{L}earning), which outperforms state-of-the-art competitors on the M-BEIR benchmark by a large margin in supervised settings, and also exihibits strong zero-shot performance on several tasks such as composed image retrieval and text-to-video retrieval. These results underscore the generalization potential of our framework across various embedding-based retrieval tasks. Code is available at this https URL
Abstract (translated)
通用多模态检索（UMR）旨在解决复杂检索任务，其中查询和候选对象跨越多种不同的模式。近年来，随着多模态大规模语言模型（MLLMs）的出现，这一领域取得了显著进展。尽管文献中基于MLLM的方法在顶尖水平上广泛采用对比学习原理，但它们的具体训练方法却各不相同。虽然这些方法已经取得了一定的成功，但其检索能力背后的机制仍很大程度上未被探索，这可能导致了次优性能和有限的泛化能力。 为了解决这些问题，我们提出一项全面研究，旨在揭示使用MLLM进行UMR有效嵌入学习的关键因素。首先，我们将实现一个通用的基于MLLM的嵌入学习流水线，并系统地分析促进高性能通用检索系统的主因。在此基础上，我们会探索嵌入生成和训练策略各个方面的细节，包括逐步过渡、难负样本挖掘以及重排序器蒸馏等。 值得注意的是，我们的研究发现一些经常被忽视的因素对模型性能有着显著影响。基于这些发现，我们提出了一种统一的框架——U-MARVEL（通用多模态检索嵌入学习），该框架在监督设置下于M-BEIR基准测试中大幅超越现有最佳竞争者，并且还在诸如组合图像检索和文本到视频检索等任务上表现出强大的零样本性能。这些结果凸显了我们的框架跨各种基于嵌入的检索任务上的泛化潜力。 相关代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2507.14902
