Abstract
This thesis investigates two key phenomena in large language models (LLMs): in-context learning (ICL) and model collapse. We study ICL in a linear transformer with tied weights trained on linear regression tasks, and show that minimising the in-context loss leads to a phase transition in the learned parameters. Above a critical context length, the solution develops a skew-symmetric component. We prove this by reducing the forward pass of the linear transformer under weight tying to preconditioned gradient descent, and then analysing the optimal preconditioner. This preconditioner includes a skew-symmetric component, which induces a rotation of the gradient direction. For model collapse, we use martingale and random walk theory to analyse simplified settings - linear regression and Gaussian fitting - under both replacing and cumulative data regimes. We strengthen existing results by proving almost sure convergence, showing that collapse occurs unless the data grows sufficiently fast or is retained over time. Finally, we introduce the notion of context collapse: a degradation of context during long generations, especially in chain-of-thought reasoning. This concept links the dynamics of ICL with long-term stability challenges in generative models.
Abstract (translated)
这篇论文探讨了大型语言模型（LLM）中的两个关键现象：上下文学习（ICL）和模型崩溃。我们通过在训练线性回归任务的线性变压器中研究ICL，发现最小化上下文损失会导致学习参数发生相变。当上下文长度超过临界值时，解决方案会发展出一个斜对称分量。我们通过将权重绑定的线性变换器的前向传递简化为预条件梯度下降，并分析最优预条件器来证明这一点。该预条件器包含了一个斜对称分量，这会导致梯度方向发生旋转。 对于模型崩溃现象，我们利用鞅和随机游走理论在两种简化的设置下进行研究：线性回归和高斯拟合，在替换数据和累积数据的规则下。通过证明几乎确定收敛来加强现有结果，表明除非数据增长足够快或随时间保留，否则会发生崩溃。 最后，本文引入了上下文崩溃的概念：在长时间生成过程中，特别是链式思维推理中的上下文质量退化现象。这个概念将ICL的动力学与生成模型长期稳定性挑战联系起来。
URL
https://arxiv.org/abs/2601.00923
