Abstract
Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, multi-modal search and recommendation, and retrieval-augmented generation (RAG). To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering - spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings.
Abstract (translated)
多模态嵌入模型在各种下游任务中发挥了关键作用，如语义相似性、信息检索和跨不同模式的聚类。然而，现有的多模态嵌入方法（例如VLM2Vec、E5-V、GME）主要关注自然图像，对视频和其他视觉形式的支持有限，这限制了它们在现实场景中的应用范围，包括AI代理、多模态搜索和推荐以及检索增强生成(RAG)。为填补这一空白，我们提出了VLM2Vec-V2，这是一个用于跨多种视觉形式学习嵌入的统一框架。 首先，我们介绍MMEB-V2，这是一个综合基准测试库，它在原有的MMEB基础上新增了五种任务类型：视觉文档检索、视频检索、时间定位（temporal grounding）、视频分类和视频问答——这些任务涵盖了文本、图像、视频和可视文档输入。接下来，我们训练了一个通用的嵌入模型VLM2Vec-V2，该模型支持文本、图像、视频和可视文档输入。 经过广泛的实验验证，结果显示VLM2Vec-V2不仅在新引入的视频和文档检索任务中表现出色，在原有的图像基准测试上也超越了先前的方法。通过全面评估，我们的研究为各种多模态嵌入模型的泛化能力提供了见解，并突出了统一嵌入学习的有效策略，从而为科研界和实际应用中的可扩展性和适应性表示学习奠定了基础。
URL
https://arxiv.org/abs/2507.04590
