Abstract
Large language models (LLMs) have demonstrated strong performance on a variety of natural language processing (NLP) tasks. However, they often struggle with long-text sequences due to the ``lost in the middle'' phenomenon. This issue has been shown to arise from a U-shaped attention bias, where attention is disproportionately focused on the beginning and end of a text, leaving the middle section underrepresented. While previous studies have attributed this bias to position encoding, our research first identifies an additional factor: initial saliency. It means that in the attention computation for each token, tokens with higher attention weights relative to the initial token tend to receive more attention in the prediction of the next token. We further find that utilizing this property by scaling attention weight between the initial token and others improves the model's ability to process long contexts, achieving a maximum improvement of 3.6\% in MDQA dataset. Moreover, combining this approach with existing methods to reduce position encoding bias further enhances performance, achieving a maximum improvement of 3.4\% in KV-Retrieval tasks.
Abstract (translated)
大型语言模型（LLMs）在多种自然语言处理（NLP）任务中表现出强大的性能。然而，它们通常难以处理长文本序列，这是因为所谓的“中间部分丢失”现象。这个问题已被证明是由U形注意力偏置引起的，即注意力过于集中在文本的开头和结尾部分，而忽略了中间部分。尽管之前的研究将这种偏差归因于位置编码的影响，我们的研究首次识别出另一个重要因素：初始显著性。这意味着，在每个令牌的注意计算中，相对于初始令牌具有更高注意权重的令牌在预测下一个令牌时会获得更多的关注。 我们进一步发现，通过调整初始令牌与其他令牌之间的注意力权重来利用这一特性，可以增强模型处理长上下文的能力，并在MDQA数据集中实现了最高3.6%的性能提升。此外，将这种方法与现有的减少位置编码偏差的方法相结合，还可以进一步提高性能，在KV-Retrieval任务中实现最高的3.4%改进。
URL
https://arxiv.org/abs/2512.13109
