Abstract
Traditional information extraction systems face challenges with text only language models as it does not consider infographics (visual elements of information) such as tables, charts, images etc. often used to convey complex information to readers. Multimodal LLM (MLLM) face challenges of finding needle in the haystack problem i.e., either longer context length or substantial number of documents as search space. Late interaction mechanism over visual language models has shown state of the art performance in retrieval-based vision augmented Q&A tasks. There are yet few challenges using it for RAG based multi-modal Q&A. Firstly, many popular and widely adopted vector databases do not support native multi-vector retrieval. Secondly, late interaction requires computation which inflates space footprint and can hinder enterprise adoption. Lastly, the current state of late interaction mechanism does not leverage the approximate neighbor search indexing methods for large speed ups in retrieval process. This paper explores a pragmatic approach to make vision retrieval process scalable and efficient without compromising on performance quality. We propose multi-step custom implementation utilizing widely adopted hybrid search (metadata & embedding) and state of the art late interaction re-ranker to retrieve best matching pages. Finally, MLLM are prompted as reader to generate answers from contextualized best matching pages. Through experiments, we observe that the proposed design is scalable (significant speed up) and stable (without degrading performance quality), hence can be used as production systems at enterprises.
Abstract (translated)
传统信息抽取系统在只处理文本的语言模型方面面临挑战，因为它们不考虑用于传达复杂信息的图形元素（如表格、图表和图像等）。多模态大型语言模型（MLLM）则面临着“大海捞针”的问题，即需要更长的上下文长度或大量的文档作为搜索空间。视觉语言模型中的后期互动机制在基于检索的视觉增强问答任务中表现出色，但在用于RAG（Retrieval-Augmented Generation）多模态问答时仍面临一些挑战。首先，许多流行的向量数据库不支持原生的多向量检索。其次，后期互动需要计算资源，这会增加存储空间占用并可能阻碍企业采用。最后，当前的后期互动机制尚未利用近似邻居搜索索引方法来大幅提升检索过程的速度。 本文探索了一种实用的方法，以使视觉检索过程既可扩展又高效，同时不牺牲性能质量。我们提出一个多步骤的自定义实现方案，结合广泛使用的混合搜索（元数据和嵌入）技术以及最新的后期互动重排器，以获取最佳匹配页面。最后，MLLM被用作读者，从上下文化的最佳匹配页面中生成答案。 通过实验观察到，所提出的架构是可扩展的（显著加速）、稳定的（不降低性能质量），因此可以作为企业的生产系统使用。
URL
https://arxiv.org/abs/2507.12378
