Abstract
The issue of hallucinations is a prevalent concern in existing Large Vision-Language Models (LVLMs). Previous efforts have primarily focused on investigating object hallucinations, which can be easily alleviated by introducing object detectors. However, these efforts neglect hallucinations in inter-object relationships, which is essential for visual comprehension. In this work, we introduce R-Bench, a novel benchmark for evaluating Vision Relationship Hallucination. R-Bench features image-level questions that focus on the existence of relationships and instance-level questions that assess local visual comprehension. We identify three types of relationship co-occurrences that lead to hallucinations: relationship-relationship, subject-relationship, and relationship-object. The visual instruction tuning dataset's long-tail distribution significantly impacts LVLMs' understanding of visual relationships. Furthermore, our analysis reveals that current LVLMs tend to disregard visual content and overly rely on the common sense knowledge of Large Language Models. They also struggle with reasoning about spatial relationships based on contextual information.
Abstract (translated)
幻觉问题一直是现有大型视觉语言模型（LVLMs）中的一个普遍关注点。之前的努力主要集中在研究物体幻觉，通过引入物体检测器可以轻松缓解这种幻觉。然而，这些努力忽视了物体关系幻觉，这对于视觉理解是至关重要的。在这项工作中，我们引入了R-Bench，一种用于评估视觉关系幻觉的新基准。R-Bench包括针对关系和实例水平的图像级问题，这些问题关注关系的存在以及评估局部视觉理解。我们确定了三种导致幻觉的关系共现类型：关系关系、主体关系和关系物体。视觉指令调整数据集的长尾分布显著影响了LVLMs对视觉关系的理解。此外，我们的分析发现，当前的LVLMs往往忽视视觉内容，过于依赖常识知识的大型语言模型。他们还难以根据上下文信息进行空间关系推理。
URL
https://arxiv.org/abs/2406.16449
