Abstract
Multimodal large language models (MLLMs) achieve strong performance on benchmarks that evaluate text, image, or video understanding separately. However, these settings do not assess a critical real-world requirement, which involves retrieving relevant evidence from large, heterogeneous multimodal corpora prior to reasoning. Most existing benchmarks restrict retrieval to small, single-modality candidate sets, substantially simplifying the search space and overstating end-to-end reliability. To address this gap, we introduce MultiHaystack, the first benchmark designed to evaluate both retrieval and reasoning under large-scale, cross-modal conditions. MultiHaystack comprises over 46,000 multimodal retrieval candidates across documents, images, and videos, along with 747 open yet verifiable questions. Each question is grounded in a unique validated evidence item within the retrieval pool, requiring evidence localization across modalities and fine-grained reasoning. In our study, we find that models perform competitively when provided with the corresponding evidence, but their performance drops sharply when required to retrieve that evidence from the full corpus. Additionally, even the strongest retriever, E5-V, achieves only 40.8% Recall@1, while state-of-the-art MLLMs such as GPT-5 experience a significant drop in reasoning accuracy from 80.86% when provided with the corresponding evidence to 51.4% under top-5 retrieval. These results indicate that multimodal retrieval over heterogeneous pools remains a primary bottleneck for MLLMs, positioning MultiHaystack as a valuable testbed that highlights underlying limitations obscured by small-scale evaluations and promotes retrieval-centric advances in multimodal systems.
Abstract (translated)
多模态大型语言模型（MLLMs）在评估文本、图像或视频理解的基准测试中表现出色。然而，这些设置未能评价一个关键的实际需求：即从大规模、异质化的多模态语料库中检索相关证据前进行推理的需求。现有的大多数基准测试限制了检索范围为小型单一模态候选集，这大大简化了搜索空间，并夸大了端到端的可靠性。为了填补这一空白，我们引入了MultiHaystack，这是首个旨在评估大规模跨模态条件下的检索和推理能力的基准。 MultiHaystack包含超过46,000个跨文档、图像和视频的多模态检索候选对象，以及747个开放且可验证的问题。每个问题都基于检索池中的一个独特的已验证证据项，需要跨模式定位证据，并进行细致推理。在我们的研究中，我们发现当模型被提供相应证据时表现良好，但它们从整个语料库中检索这些证据的能力下降幅度很大。此外，即使最强的检索器E5-V也仅实现了40.8%的召回率@1（Recall@1），而像GPT-5这样的最先进的MLLM在获得对应证据的情况下推理准确率为80.86%，但在最接近的五个检索结果中这个数值下降到了51.4%。这些结果显示，多模态异构池中的检索仍然是MLLM的主要瓶颈，并且将MultiHaystack定位为一个有价值的测试平台，揭示了由小规模评估所掩盖的基础限制，并促进了以检索为中心的多模态系统的发展。
URL
https://arxiv.org/abs/2603.05697
