Abstract
Audio-Visual Large Language Models (AVLLMs) are emerging as unified interfaces to multimodal perception. We present the first mechanistic interpretability study of AVLLMs, analyzing how audio and visual features evolve and fuse through different layers of an AVLLM to produce the final text outputs. We find that although AVLLMs encode rich audio semantics at intermediate layers, these capabilities largely fail to surface in the final text generation when audio conflicts with vision. Probing analyses show that useful latent audio information is present, but deeper fusion layers disproportionately privilege visual representations that tend to suppress audio cues. We further trace this imbalance to training: the AVLLM's audio behavior strongly matches its vision-language base model, indicating limited additional alignment to audio supervision. Our findings reveal a fundamental modality bias in AVLLMs and provide new mechanistic insights into how multimodal LLMs integrate audio and vision.
Abstract (translated)
视听大语言模型（AVLLMs）正逐渐成为多模态感知的统一接口。我们首次对AVLLMs进行了机制可解释性研究，分析了音频和视觉特征如何通过AVLLM的不同层演变与融合，以产生最终文本输出。我们发现，尽管AVLLMs在中间层编码了丰富的音频语义信息，但当音频与视觉信息冲突时，这些能力在最终文本生成中 largely 未能显现。探测分析表明，有用的潜在音频信息确实存在，但深层融合层过度偏向视觉表征，从而倾向于压制音频线索。我们进一步将这种不平衡追溯至训练过程：AVLLM的音频行为与其视觉-语言基础模型高度匹配，表明其对音频监督的额外对齐有限。我们的研究揭示了AVLLMs中存在的根本性模态偏差，并为多模态大语言模型如何整合音频与视觉提供了新的机制性见解。
URL
https://arxiv.org/abs/2604.02605
