Abstract
Effective multimodal reasoning depends on the alignment of visual and linguistic representations, yet the mechanisms by which vision-language models (VLMs) achieve this alignment remain poorly understood. We introduce a methodological framework that deliberately maintains a frozen large language model (LLM) and a frozen vision transformer (ViT), connected solely by training a linear adapter during visual instruction tuning. This design is fundamental to our approach: by keeping the language model frozen, we ensure it maintains its original language representations without adaptation to visual data. Consequently, the linear adapter must map visual features directly into the LLM's existing representational space rather than allowing the language model to develop specialized visual understanding through fine-tuning. Our experimental design uniquely enables the use of pre-trained sparse autoencoders (SAEs) of the LLM as analytical probes. These SAEs remain perfectly aligned with the unchanged language model and serve as a snapshot of the learned language feature-representations. Through systematic analysis of SAE reconstruction error, sparsity patterns, and feature SAE descriptions, we reveal the layer-wise progression through which visual representations gradually align with language feature representations, converging in middle-to-later layers. This suggests a fundamental misalignment between ViT outputs and early LLM layers, raising important questions about whether current adapter-based architectures optimally facilitate cross-modal representation learning.
Abstract (translated)
有效的多模态推理依赖于视觉和语言表示的对齐，然而，视觉-语言模型（VLMs）如何实现这种对齐的机制仍然不甚明了。我们引入了一种方法论框架，在该框架中特意保持大规模语言模型（LLM）和视觉变压器（ViT）冻结状态，并仅通过训练线性适配器在视觉指令微调过程中进行连接。这一设计是我们的方法的基础：通过保持语言模型冻结，确保它保留其原有的语言表示而不适应于视觉数据的调整。因此，线性适配器必须将视觉特征直接映射到LLM已有的表示空间中，而不是允许语言模型通过微调发展专门的视觉理解。 我们实验设计的独特之处在于能够使用预训练的稀疏自编码器（SAEs）作为分析探针，这些SAE与未改变的语言模型完全对齐，并且反映了学习到的语言特征表示。通过对SAE重构误差、稀疏模式和特征SAE描述进行系统的分析，我们揭示了视觉表征如何逐层进展，逐渐与语言特征表示对齐，并在中后期层级收敛。这表明ViT输出与早期LLM层之间存在基本不对齐的问题，引发了关于当前基于适配器的架构是否能有效促进跨模态表示学习的重要问题。
URL
https://arxiv.org/abs/2506.11976
