Abstract
Counterfactual reasoning, as a crucial manifestation of human intelligence, refers to making presuppositions based on established facts and extrapolating potential outcomes. Existing multimodal large language models (MLLMs) have exhibited impressive cognitive and reasoning capabilities, which have been examined across a wide range of Visual Question Answering (VQA) benchmarks. Nevertheless, how will existing MLLMs perform when faced with counterfactual questions? To answer this question, we first curate a novel \textbf{C}ounter\textbf{F}actual \textbf{M}ulti\textbf{M}odal reasoning benchmark, abbreviated as \textbf{CFMM}, to systematically assess the counterfactual reasoning capabilities of MLLMs. Our CFMM comprises six challenging tasks, each including hundreds of carefully human-labeled counterfactual questions, to evaluate MLLM's counterfactual reasoning capabilities across diverse aspects. Through experiments, interestingly, we find that existing MLLMs prefer to believe what they see, but ignore the counterfactual presuppositions presented in the question, thereby leading to inaccurate responses. Furthermore, we evaluate a wide range of prevalent MLLMs on our proposed CFMM. The significant gap between their performance on our CFMM and that on several VQA benchmarks indicates that there is still considerable room for improvement in existing MLLMs toward approaching human-level intelligence. On the other hand, through boosting MLLMs performances on our CFMM in the future, potential avenues toward developing MLLMs with advanced intelligence can be explored.
Abstract (translated)
作为人类智力的关键表现形式，反事实推理是指根据已有的事实做出假设，并推断可能的后果。已经存在的多模态大型语言模型（MLLMs）在广泛的视觉问答基准测试中表现出了出色的认知和推理能力。然而，当它们面对反事实问题时，现有的MLLMs会表现如何？为了回答这个问题，我们首先收集了一个新的反事实推理基准，称为反事实推理多模态模型（CFMM），以系统地评估MLLMs的反事实推理能力。我们的CFMM包括六个具有挑战性的任务，每个任务包括数百个精心人类标注的反事实问题，以评估MLLM在多样性方面的反事实推理能力。通过实验，我们发现，现有的MLLM更喜欢相信它们所看到的东西，但忽视了问题中提出的反事实假设，从而导致不准确的回答。此外，我们在CFMM上评估了各种常见的MLLM。它们在CFMM上的表现与我们CFMM上的表现之间的显著差距表明，现有MLLM在接近人类智能方面还有很大的改进空间。另一方面，通过在CFMM上提高MLLMs的表现，可以探索开发具有高级智能的MLLM的潜在途径。
URL
https://arxiv.org/abs/2404.12966
