Abstract
We study how visual artifacts introduced by diffusion-based inpainting affect language generation in vision-language models. We use a two-stage diagnostic setup in which masked image regions are reconstructed and then provided to captioning models, enabling controlled comparisons between captions generated from original and reconstructed inputs. Across multiple datasets, we analyze the relationship between reconstruction fidelity and downstream caption quality. We observe consistent associations between pixel-level and perceptual reconstruction metrics and both lexical and semantic captioning performance. Additional analysis of intermediate visual representations and attention patterns shows that inpainting artifacts lead to systematic, layer-dependent changes in model behavior. Together, these results provide a practical diagnostic framework for examining how visual reconstruction quality influences language generation in multimodal systems.
Abstract (translated)
我们研究了基于扩散的图像修复过程中引入的视觉伪影如何影响视觉-语言模型中的语言生成。为此，我们采用了一个两阶段诊断设置，在该设置中，被遮挡的图像区域会被重建并提供给描述模型，从而可以对从原始和重构输入生成的描述进行有控制的比较。在多个数据集上，我们分析了重建保真度与下游描述质量之间的关系。我们观察到像素级和感知重建指标与词汇和语义描述性能之间存在一致的相关性。对于中间视觉表示和注意力模式的进一步分析表明，修复伪影会导致模型行为发生系统性和层级相关的改变。综合起来，这些结果提供了一个实用的诊断框架，用于考察视觉重构质量如何影响多模态系统中的语言生成。
URL
https://arxiv.org/abs/2602.20520
