Abstract
Despite significant advancements in Large Vision Language Models (LVLMs), a gap remains, particularly regarding their interpretability and how they locate and interpret textual information within images. In this paper, we explore various LVLMs to identify the specific heads responsible for recognizing text from images, which we term the Optical Character Recognition Head (OCR Head). Our findings regarding these heads are as follows: (1) Less Sparse: Unlike previous retrieval heads, a large number of heads are activated to extract textual information from images. (2) Qualitatively Distinct: OCR heads possess properties that differ significantly from general retrieval heads, exhibiting low similarity in their characteristics. (3) Statically Activated: The frequency of activation for these heads closely aligns with their OCR scores. We validate our findings in downstream tasks by applying Chain-of-Thought (CoT) to both OCR and conventional retrieval heads and by masking these heads. We also demonstrate that redistributing sink-token values within the OCR heads improves performance. These insights provide a deeper understanding of the internal mechanisms LVLMs employ in processing embedded textual information in images.
Abstract (translated)
尽管大型视觉语言模型（LVLM）取得了显著进展，但在这些模型的可解释性和它们如何在图像中定位和解读文本信息方面仍存在差距。在这篇论文中，我们探索了各种LVLM以识别负责从图像中提取文本信息的具体头部，并将其称为光学字符识别头部（OCR 头）。关于这些头部的发现如下： 1. **较少稀疏**：与之前的检索头部不同，大量头部被激活来提取图像中的文本信息。 2. **定性差异显著**：OCR 头部具有与通用检索头部大不相同的特性，并且它们的特点之间相似度较低。 3. **静态激活**：这些头部的激活频率与其 OCR 分数紧密相关。 为了验证我们的发现，我们在下游任务中应用了链式思维（CoT）方法对OCR头和传统检索头进行了测试，并通过屏蔽这些头部来观察效果。我们还证明，在OCR 头部内重新分配sink-token值可以提高性能。 这些见解为理解LVLM在处理图像中的嵌入文本信息时所采用的内部机制提供了更深入的理解。
URL
https://arxiv.org/abs/2505.15865
