Abstract
Despite the remarkable capabilities of Multimodal Large Language Models (MLLMs), they still suffer from visual fading in long-context scenarios. Specifically, the attention to visual tokens diminishes as the text sequence lengthens, leading to text generation detached from visual constraints. We attribute this degradation to the inherent inductive bias of Multimodal RoPE, which penalizes inter-modal attention as the distance between visual and text tokens increases. To address this, we propose inter-modal Distance Invariant Position Encoding (DIPE), a simple but effective mechanism that disentangles position encoding based on modality interactions. DIPE retains the natural relative positioning for intra-modal interactions to preserve local structure, while enforcing an anchored perceptual proximity for inter-modal interactions. This strategy effectively mitigates the inter-modal distance-based penalty, ensuring that visual signals remain perceptually consistent regardless of the context length. Experimental results demonstrate that by integrating DIPE with Multimodal RoPE, the model maintains stable visual grounding in long-context scenarios, significantly alleviating visual fading while preserving performance on standard short-context benchmarks. Code is available at this https URL.
Abstract (translated)
尽管多模态大型语言模型（MLLM）具有显著的能力，但在长上下文场景中仍然存在视觉衰减的问题。具体而言，随着文本序列长度的增加，对视觉标记的关注度逐渐减弱，导致生成的文字与视觉约束脱钩。我们认为这种退化是由于多模态RoPE的内在归纳偏差所导致的，在视觉和文本标记之间的距离增大时，会惩罚跨模态注意力。 为了解决这个问题，我们提出了跨模态距离不变位置编码（DIPE），这是一种简单但有效的机制，通过基于模态交互来分离位置编码。DIPE保留了单个模态内部互动中的自然相对定位以保持局部结构，并在跨模态互动中强制执行感知上的固定接近度。这一策略有效地缓解了基于跨模态距离的惩罚问题，确保视觉信号无论上下文长度如何都能保持感知上的一致性。 实验结果显示，通过将DIPE与多模态RoPE集成在一起，模型能够在长上下文中维持稳定的视觉定位，并显著减轻视觉衰减，同时在标准短上下文基准测试中保持性能。相关代码可在该链接获取：[此URL]（请使用实际的网址替代"this https URL"）。
URL
https://arxiv.org/abs/2603.10863
