Abstract
Existing Large Vision-Language Models (LVLMs) primarily align image features of vision encoder with Large Language Models (LLMs) to leverage their superior text generation capabilities. However, the scale disparity between vision encoder and language model may led to LLMs assuming a predominant role in multi-modal comprehension. This imbalance in LVLMs may result in the instances of hallucinatory. Concretely, LVLMs may generate consistent descriptions with or without visual input, indicating that certain outputs are influenced solely by context text. We refer to this phenomenon as "text inertia." To counteract this issue, we introduce a training-free algorithm to find an equilibrium point between image comprehension and language inference. Specifically, we adaptively involve adjusting and amplifying the attention weights assigned to image tokens, thereby granting greater prominence to visual elements. Meanwhile, we subtract the logits of multi-modal inputs from ones of pure text input, which can help LVLMs be not biased towards LLMs. By enhancing images tokens and reducing the stubborn output of LLM, we can let LVLM pay more attention to images, towards alleviating text inertia and reducing the hallucination in LVLMs. Our extensive experiments shows that this method substantially reduces the frequency of hallucinatory outputs in various LVLMs in terms of different metrics. Project page is available at this https URL.
Abstract (translated)
目前，大型视觉语言模型（LVLMs）主要将视觉编码器图像特征与大型语言模型（LLM）对齐，以利用它们在文本生成方面的卓越能力。然而，视觉编码器与语言模型之间的规模差异可能导致LLM在多模态理解中扮演主导角色。这种不平衡在LVLMs中可能导致幻觉实例的出现。具体来说，LVLMs可能生成带有或没有视觉输入的稳定描述，表明某些输出仅受上下文文本影响。我们将这种现象称为“文本惯性”。为了应对这个问题，我们引入了一种无需训练的算法，在图像理解和语言推理之间找到平衡点。具体来说，我们自适应地调整和放大分配给图像标记的注意力权重，从而赋予视觉元素更大的突出地位。同时，我们将多模态输入的logits从纯文本输入的logits中减去，这可以帮助LVLMs不会偏向LLM。通过增强图像标记并减少LLM的僵化输出，我们可以让LVLM更关注图像，减轻文本惯性和LVLMs中的幻觉。我们的广泛实验证明，这种方法在各种LVLMs上显著减少了幻觉输出的频率，从不同指标来看。项目页面可以通过这个链接获得。
URL
https://arxiv.org/abs/2407.21771
