Abstract
Multi-modal Large Language Models (MLLMs) have demonstrated remarkable performance on various visual-language understanding and generation tasks. However, MLLMs occasionally generate content inconsistent with the given images, which is known as "hallucination". Prior works primarily center on evaluating hallucination using standard, unperturbed benchmarks, which overlook the prevalent occurrence of perturbed inputs in real-world scenarios-such as image cropping or blurring-that are critical for a comprehensive assessment of MLLMs' hallucination. In this paper, to bridge this gap, we propose Hallu-PI, the first benchmark designed to evaluate Hallucination in MLLMs within Perturbed Inputs. Specifically, Hallu-PI consists of seven perturbed scenarios, containing 1,260 perturbed images from 11 object types. Each image is accompanied by detailed annotations, which include fine-grained hallucination types, such as existence, attribute, and relation. We equip these annotations with a rich set of questions, making Hallu-PI suitable for both discriminative and generative tasks. Extensive experiments on 12 mainstream MLLMs, such as GPT-4V and Gemini-Pro Vision, demonstrate that these models exhibit significant hallucinations on Hallu-PI, which is not observed in unperturbed scenarios. Furthermore, our research reveals a severe bias in MLLMs' ability to handle different types of hallucinations. We also design two baselines specifically for perturbed scenarios, namely Perturbed-Reminder and Perturbed-ICL. We hope that our study will bring researchers' attention to the limitations of MLLMs when dealing with perturbed inputs, and spur further investigations to address this issue. Our code and datasets are publicly available at this https URL.
Abstract (translated)
多模态大型语言模型（MLLMs）在各种视觉语言理解和生成任务上表现出了非凡的性能。然而，MLLMs偶尔生成的内容与给定图像不一致，这被称为“虚构”。之前的工作主要集中在使用标准、无扰动的基准来评估虚构，而忽视了现实场景中普遍存在的对输入的扰动-例如图像裁剪或模糊-这对于全面评估MLLMs的虚构能力至关重要。在本文中，为了填补这个空白，我们提出了Hallu-PI，第一个专门针对MLLMs在扰动输入评估虚构的基准。具体来说，Hallu-PI包括7个扰动场景，包含11个物体类型的1260个扰动图像。每个图像都配有详细的注释，包括细粒度的虚构类型，如存在、属性和关系。我们将这些注释与丰富的问题集相结合，使Hallu-PI适用于区分和生成任务。对12个主流MLLM（如GPT-4V和Gemini-Pro Vision）的广泛实验表明，这些模型在Hallu-PI上表现出显著的虚构，而在无扰动场景中并未观察到。此外，我们的研究揭示了MLLM处理不同类型的虚构能力的严重偏见。我们还针对扰动场景设计了两组基线，即Perturbed-Reminder和Perturbed-ICL。我们希望我们的研究能够吸引研究人员的注意，使他们意识到MLLM在处理扰动输入方面的局限性，并进一步研究以解决这个问题。我们的代码和数据集可以在本文的https:// URL上公开获取。
URL
https://arxiv.org/abs/2408.01355
