Abstract
Understanding the interplay between intra-modality dependencies (the contribution of an individual modality to a target task) and inter-modality dependencies (the relationships between modalities and the target task) is fundamental to advancing multi-modal learning. However, the nature of and interaction between these dependencies within current benchmark evaluations remains poorly characterized. In this work, we present a large-scale empirical study to quantify these dependencies across 23 visual question-answering benchmarks using multi-modal large language models (MLLMs) covering domains such as general and expert knowledge reasoning, optical character recognition, and document understanding. Our findings show that the reliance on vision, question (text), and their interaction varies significantly, both across and within benchmarks. We discover that numerous benchmarks intended to mitigate text-only biases have inadvertently amplified image-only dependencies. This characterization persists across model sizes, as larger models often use these intra-modality dependencies to achieve high performance that mask an underlying lack of multi-modal reasoning. We provide a quantitative characterization of multi-modal datasets, enabling a principled approach to multi-modal benchmark design and evaluation.
Abstract (translated)
理解模态内部依赖（即单一模态对目标任务的贡献）与跨模态依赖（即不同模态之间以及它们与目标任务之间的关系）对于推进多模态学习至关重要。然而，当前基准评估中这些依赖性及其相互作用的本质仍然未被充分描述。在这项工作中，我们进行了一次大规模实证研究，旨在通过覆盖通用和专家知识推理、光学字符识别及文档理解等领域的多模态大型语言模型（MLLMs），量化23个视觉问答基准中的这些依赖关系。我们的发现表明，在不同基准之间以及同一基准内部，对视觉信息、问题文本及其相互作用的依赖性存在显著差异。我们还发现，许多旨在缓解仅基于文本偏差的设计，反而无意中放大了图像单一模态的依赖性。这种特征在各种模型大小中都普遍存在，大型模型往往利用这些单模态依赖来实现高表现，从而掩盖其内部缺乏多模态推理能力的问题。 我们的研究提供了对多模态数据集的量化描述，为进一步制定和评估多模态基准测试提供了一个原则性的方法论。
URL
https://arxiv.org/abs/2509.23499
