Abstract
Next-token prediction (NTP) is the cornerstone of modern large language models (LLMs) pretraining, driving their unprecedented capabilities in text generation, reasoning, and instruction following. However, the token-level prediction limits the model's capacity to capture higher-level semantic structures and long-range contextual relationships. To overcome this limitation, we introduce \textbf{ContextLM}, a framework that augments standard pretraining with an inherent \textbf{next-context prediction} objective. This mechanism trains the model to learn predictive representations of multi-token contexts, leveraging error signals derived from future token chunks. Crucially, ContextLM achieves this enhancement while remaining fully compatible with the standard autoregressive, token-by-token evaluation paradigm (e.g., perplexity). Extensive experiments on the GPT2 and Pythia model families, scaled up to $1.5$B parameters, show that ContextLM delivers consistent improvements in both perplexity and downstream task performance. Our analysis indicates that next-context prediction provides a scalable and efficient pathway to stronger language modeling, yielding better long-range coherence and more effective attention allocation with minimal computational overhead.
Abstract (translated)
下一代标记预测（NTP）是现代大型语言模型（LLM）预训练的基础，推动了这些模型在文本生成、推理和指令遵循方面的前所未有的能力。然而，基于标记级别的预测限制了模型捕捉更高层次的语义结构以及长距离上下文关系的能力。为了解决这一局限性，我们引入了**ContextLM**框架，该框架通过添加内在的**下一个上下文预测**目标来增强标准预训练过程。这种机制使模型能够学习多令牌上下文的预测表示，并利用来自未来标记块的误差信号。至关重要的是，ContextLM能够在完全兼容标准自回归、逐令牌评估范式（如困惑度）的同时实现这一改进。在GPT2和Pythia模型系列上的广泛实验表明，ContextLM可以在参数规模达到15亿的情况下，在困惑度和下游任务性能上提供一致的改进。我们的分析表明，上下文预测为更强的语言建模提供了可扩展且高效的途径，能够带来更好的长距离连贯性和更有效的注意力分配，并且几乎不需要额外计算资源。
URL
https://arxiv.org/abs/2510.20280
