Abstract
The scaling law, a cornerstone of Large Language Model (LLM) development, predicts improvements in model performance with increasing computational resources. Yet, while empirically validated, its theoretical underpinnings remain poorly understood. This work formalizes the learning dynamics of transformer-based language models as an ordinary differential equation (ODE) system, then approximates this process to kernel behaviors. Departing from prior toy-model analyses, we rigorously analyze stochastic gradient descent (SGD) training for multi-layer transformers on sequence-to-sequence data with arbitrary data distribution, closely mirroring real-world conditions. Our analysis characterizes the convergence of generalization error to the irreducible risk as computational resources scale with data, especially during the optimization process. We establish a theoretical upper bound on excess risk characterized by a distinct phase transition. In the initial optimization phase, the excess risk decays exponentially relative to the computational cost ${\sf C}$. However, once a specific resource allocation threshold is crossed, the system enters a statistical phase, where the generalization error follows a power-law decay of $\Theta(\mathsf{C}^{-1/6})$. Beyond this unified framework, our theory derives isolated scaling laws for model size, training time, and dataset size, elucidating how each variable independently governs the upper bounds of generalization.
Abstract (translated)
缩放法则，大型语言模型（LLM）开发的基石，预测了随着计算资源增加而改进模型性能的趋势。尽管这一理论已经在实证研究中得到了验证，但其背后的理论基础仍然不甚明了。这项工作将基于变压器的语言模型的学习动力学形式化为一个常微分方程（ODE）系统，并将其近似到核行为中去。不同于以往的玩具模型分析，我们严谨地分析了多层变压器在任意数据分布下的序列到序列数据上的随机梯度下降（SGD）训练过程，这更贴近现实世界的条件。 我们的研究描述了随着计算资源与数据量一同增加时泛化误差收敛至不可还原的风险的过程。并且，在优化过程中建立了由一个独特的相变特征定义的理论风险上限。在初始优化阶段，超额风险相对于计算成本${\sf C}$呈指数衰减形式。然而，一旦特定的资源配置阈值被跨越后，系统进入统计相位，在此阶段，泛化误差遵循$\Theta(\mathsf{C}^{-1/6})$的幂律衰退模式。 除了这个统一框架外，我们的理论还为模型大小、训练时间以及数据集规模分别推导出了孤立缩放法则，解释了每个变量如何独立地影响泛化的上限。
URL
https://arxiv.org/abs/2512.22088
