Abstract
We present Omni-Embed-Nemotron, a unified multimodal retrieval embedding model developed to handle the increasing complexity of real-world information needs. While Retrieval-Augmented Generation (RAG) has significantly advanced language models by incorporating external knowledge, existing text-based retrievers rely on clean, structured input and struggle with the visually and semantically rich content found in real-world documents such as PDFs, slides, or videos. Recent work such as ColPali has shown that preserving document layout using image-based representations can improve retrieval quality. Building on this, and inspired by the capabilities of recent multimodal models such as Qwen2.5-Omni, we extend retrieval beyond text and images to also support audio and video modalities. Omni-Embed-Nemotron enables both cross-modal (e.g., text - video) and joint-modal (e.g., text - video+audio) retrieval using a single model. We describe the architecture, training setup, and evaluation results of Omni-Embed-Nemotron, and demonstrate its effectiveness in text, image, and video retrieval.
Abstract (translated)
我们介绍了Omni-Embed-Nemotron，这是一种统一的多模态检索嵌入模型，旨在处理日益复杂的真实世界信息需求。虽然检索增强生成（RAG）通过整合外部知识显著提升了语言模型的能力，但现有的基于文本的检索器依赖于干净、结构化的输入，并且在处理如PDF文件、幻灯片或视频中发现的视觉和语义丰富的内容时显得力不从心。最近的研究，例如ColPali，表明使用图像表示来保留文档布局可以提升检索质量。在此基础上，并受到近期多模态模型（如Qwen2.5-Omni）能力的启发，我们将检索范围扩展到不仅包括文本和图像，还包括音频和视频模式。Omni-Embed-Nemotron能够同时支持跨模态（例如，文本与视频）和联合模态（例如，文本、视频加音频）检索，仅使用单一模型即可实现。 本文描述了Omni-Embed-Nemotron的架构、训练设置及评估结果，并展示了其在文本、图像和视频检索中的有效性。
URL
https://arxiv.org/abs/2510.03458
