Abstract
Universal multimodal embedding models play a critical role in tasks such as interleaved image-text retrieval, multimodal RAG, and multimodal clustering. However, our empirical results indicate that existing LMM-based embedding models trained with the standard InfoNCE loss exhibit a high degree of overlap in similarity distribution between positive and negative pairs, making it challenging to distinguish hard negative pairs effectively. To deal with this issue, we propose a simple yet effective framework that dynamically improves the embedding model's representation learning for negative pairs based on their discriminative difficulty. Within this framework, we train a series of models, named LLaVE, and evaluate them on the MMEB benchmark, which covers 4 meta-tasks and 36 datasets. Experimental results show that LLaVE establishes stronger baselines that achieve state-of-the-art (SOTA) performance while demonstrating strong scalability and efficiency. Specifically, LLaVE-2B surpasses the previous SOTA 7B models, while LLaVE-7B achieves a further performance improvement of 6.2 points. Although LLaVE is trained on image-text data, it can generalize to text-video retrieval tasks in a zero-shot manner and achieve strong performance, demonstrating its remarkable potential for transfer to other embedding tasks.
Abstract (translated)
通用多模态嵌入模型在交错的图像-文本检索、多模态RAG（Retrieval-Augmented Generation）和多模态聚类等任务中扮演着关键角色。然而，我们的实证研究表明，基于现有LMM（Large Multimodal Model）并使用标准InfoNCE损失函数训练的嵌入模型，在正负对之间的相似性分布存在高度重叠的问题，这使得有效区分难例负样本变得困难。 为了解决这一问题，我们提出了一种简单而有效的框架，该框架能够根据负样本的判别难度动态改进嵌入模型的表现学习能力。在这一框架内，我们训练了一系列模型，命名为LLaVE，并在覆盖4个元任务和36个数据集的MMEB基准上对其进行了评估。实验结果显示，LLaVE建立了更强的基础模型，在实现SOTA（State-of-the-Art）性能的同时还展示了强大的扩展性和效率。具体来说，LLaVE-2B超过了之前的7B SOTA模型，而LLaVE-7B则进一步实现了6.2分的性能提升。 尽管LLaVE是基于图像-文本数据进行训练的，但它可以以零样本的方式推广到文本-视频检索任务，并取得强大的性能表现，这表明它在转移到其他嵌入任务中具有显著潜力。
URL
https://arxiv.org/abs/2503.04812
