Abstract
Multimodal retrieval is the task of aggregating information from queries across heterogeneous modalities to retrieve desired targets. State-of-the-art multimodal retrieval models can understand complex queries, yet they are typically limited to two modalities: text and vision. This limitation impedes the development of universal retrieval systems capable of comprehending queries that combine more than two modalities. To advance toward this goal, we present OmniRet, the first retrieval model capable of handling complex, composed queries spanning three key modalities: text, vision, and audio. Our OmniRet model addresses two critical challenges for universal retrieval: computational efficiency and representation fidelity. First, feeding massive token sequences from modality-specific encoders to Large Language Models (LLMs) is computationally inefficient. We therefore introduce an attention-based resampling mechanism to generate compact, fixed-size representations from these sequences. Second, compressing rich omni-modal data into a single embedding vector inevitably causes information loss and discards fine-grained details. We propose Attention Sliced Wasserstein Pooling to preserve these fine-grained details, leading to improved omni-modal representations. OmniRet is trained on an aggregation of approximately 6 million query-target pairs spanning 30 datasets. We benchmark our model on 13 retrieval tasks and a MMEBv2 subset. Our model demonstrates significant improvements on composed query, audio and video retrieval tasks, while achieving on-par performance with state-of-the-art models on others. Furthermore, we curate a new Audio-Centric Multimodal Benchmark (ACM). This new benchmark introduces two critical, previously missing tasks-composed audio retrieval and audio-visual retrieval to more comprehensively evaluate a model's omni-modal embedding capacity.
Abstract (translated)
多模态检索的任务是从异构模式的查询中聚合信息，以检索出所需的目标。当前最先进的多模态检索模型能够理解复杂的查询，但它们通常仅限于处理两种模式：文本和视觉。这种限制阻碍了开发能够理解结合超过两种模式的查询的通用检索系统的发展。为了朝着这一目标迈进，我们提出了OmniRet——首个能够处理跨越三种关键模式（文本、视觉和音频）的复杂组合查询的检索模型。我们的OmniRet模型解决了通用检索面临的两个重要挑战：计算效率和表示保真度。 首先，将大量来自特定模式编码器的标记序列输入大型语言模型（LLMs）在计算上是低效的。因此，我们引入了一种基于注意力机制的重采样机制来从这些序列中生成紧凑且固定大小的表示形式。其次，将丰富的跨模态数据压缩成单个嵌入向量不可避免地会导致信息丢失，并丢弃细粒度的细节。为此，我们提出了注意切片瓦瑟斯坦池化（Attention Sliced Wasserstein Pooling）以保留这些细粒度的细节，从而提高跨模态表示的质量。 OmniRet是在一个包含约600万查询-目标对的数据集汇总上进行训练的，这些数据横跨30个不同的数据集。我们在13个检索任务和MMEBv2的一个子集上基准测试了我们的模型。在组合查询、音频和视频检索任务中，我们的模型显示出了显著改进，而在其他任务上的表现与当前最先进水平相当。 此外，我们还策划了一个新的以音频为中心的多模态基准（Audio-Centric Multimodal Benchmark, ACM）。这个新基准引入了两个关键且之前缺失的任务：组合音频检索和音视频检索。这两个任务旨在更全面地评估模型在跨模式嵌入能力方面的表现。
URL
https://arxiv.org/abs/2603.02098
