Abstract
In this paper, we provide the first precise distributional characterization of gradient descent iterates for general multi-layer neural networks under the canonical single-index regression model, in the `finite-width proportional regime' where the sample size and feature dimension grow proportionally while the network width and depth remain bounded. Our non-asymptotic state evolution theory captures Gaussian fluctuations in first-layer weights and concentration in deeper-layer weights, and remains valid for non-Gaussian features. Our theory differs from existing neural tangent kernel (NTK), mean-field (MF) theories and tensor program (TP) in several key aspects. First, our theory operates in the finite-width regime whereas these existing theories are fundamentally infinite-width. Second, our theory allows weights to evolve from individual initializations beyond the lazy training regime, whereas NTK and MF are either frozen at or only weakly sensitive to initialization, and TP relies on special initialization schemes. Third, our theory characterizes both training and generalization errors for general multi-layer neural networks beyond the uniform convergence regime, whereas existing theories study generalization almost exclusively in two-layer settings. As a statistical application, we show that vanilla gradient descent can be augmented to yield consistent estimates of the generalization error at each iteration, which can be used to guide early stopping and hyperparameter tuning. As a further theoretical implication, we show that despite model misspecification, the model learned by gradient descent retains the structure of a single-index function with an effective signal determined by a linear combination of the true signal and the initialization.
Abstract (translated)
在这篇论文中，我们首次提供了多层神经网络在标准单索引回归模型下的梯度下降迭代的精确分布特征，在“有限宽度比例制度”下进行研究，即样本数量和特征维度按比例增长的同时，网络的宽度和深度保持不变。我们的非渐近状态演化理论捕捉了第一层权重中的高斯波动和更深层级权重中的集中特性，并且对非高斯特征依然有效。 该理论在几个关键方面与现有的神经切片核（NTK）、均值场（MF）理论以及张量程序（TP）有所不同。首先，我们的理论适用于有限宽度的情形，而这些现有理论本质上是无限宽度的。其次，我们的理论允许权重从个体初始化演变超出懒惰训练阶段，而NTK和MF要么在初始化时冻结不变，要么对初始化仅有微弱敏感性，并且TP依赖于特殊的初始化方案。第三，我们的理论描述了广泛多层神经网络的训练误差和泛化误差，超出了均匀收敛范围，而现有的理论几乎完全专注于两层设置下的泛化研究。 作为统计学的应用实例，我们展示了普通的梯度下降可以被增强以产生一致性的估计值，这些估计值在每次迭代时反映了泛化误差，并可用于指导提前停止和超参数调整。此外，从理论上讲，即使存在模型规格不当的情况，由梯度下降学习到的模型仍保留了单索引函数的形式，其有效信号由真实信号与初始化的线性组合决定。
URL
https://arxiv.org/abs/2505.04898
