Abstract
Vision-language models (VLMs) can read text from images, but where does this optical character recognition (OCR) information enter the language processing stream? We investigate the OCR routing mechanism across three architecture families (Qwen3-VL, Phi-4, InternVL3.5) using causal interventions. By computing activation differences between original images and text-inpainted versions, we identify architecture-specific OCR bottlenecks whose dominant location depends on the vision-language integration strategy: DeepStack models (Qwen) show peak sensitivity at mid-depth (about 50%) for scene text, while single-stage projection models (Phi-4, InternVL) peak at early layers (6-25%), though the exact layer of maximum effect varies across datasets. The OCR signal is remarkably low-dimensional: PC1 captures 72.9% of variance. Crucially, principal component analysis (PCA) directions learned on one dataset transfer to others, demonstrating shared text-processing pathways. Surprisingly, in models with modular OCR circuits (notably Qwen3-VL-4B), OCR removal can improve counting performance (up to +6.9 percentage points), suggesting OCR interferes with other visual processing in sufficiently modular architectures.
Abstract (translated)
视觉语言模型（VLM）可以从图像中读取文本，但光学字符识别（OCR）信息是如何进入语言处理流程的呢？我们通过因果干预的方式，在三种架构家族（Qwen3-VL、Phi-4和InternVL3.5）之间研究了OCR路由机制。通过计算原始图像与带文字填充版本之间的激活差异，我们确定了每个架构特有的OCR瓶颈，其主要位置取决于视觉语言集成策略：深层栈模型（如Qwen）在场景文本中的敏感度峰值出现在中层深度（约50%），而单阶段投影模型（Phi-4和InternVL）则在早期层次（6%-25%）达到峰值，尽管效果最大的具体层次因数据集的不同而变化。OCR信号具有非常低的维度：主成分1可以解释72.9%的变化。关键地，通过一个数据集中学习到的主要成分分析（PCA）方向可以在其他数据集中转移，证明了文本处理路径存在共享机制。出乎意料的是，在拥有模块化OCR电路的模型中（尤其是Qwen3-VL-4B），去除OCR信息可以提高计数性能（最多提升6.9个百分点），这表明在足够模块化的架构中，OCR干扰了其他视觉处理过程。
URL
https://arxiv.org/abs/2602.22918
