Abstract
A key challenge in modern deep learning theory is to explain the remarkable success of gradient-based optimization methods when training large-scale, complex deep neural networks. Though linear convergence of such methods has been proved for a handful of specific architectures, a united theory still evades researchers. This article presents a unified proof for linear convergence of continuous gradient descent, also called gradient flow, while training any neural network with piecewise non-zero polynomial activations or ReLU, sigmoid activations. Our primary contribution is a single, general theorem that not only covers architectures for which this result was previously unknown but also consolidates existing results under weaker assumptions. While our focus is theoretical and our results are only exact in the infinitesimal step size limit, we nevertheless find excellent empirical agreement between the predictions of our result and those of the practical step-size gradient descent method.
Abstract (translated)
现代深度学习理论面临的一个关键挑战是解释基于梯度的优化方法在训练大规模复杂深层神经网络时取得的成功。尽管已证明了针对少数特定架构的线性收敛，但一个统一的理论仍然困扰着研究人员。本文提出了一种对连续梯度下降（又称梯度流）进行线性收敛统一证明的方法，在使用分段非零多项式激活函数或ReLU、Sigmoid激活函数训练任何神经网络时有效。我们的主要贡献是一个单一的一般定理，该定理不仅涵盖了此前未知结果的架构类型，还以更弱的前提假设统一了现有的研究成果。尽管我们侧重于理论研究，并且我们的结论仅在微小步长极限下才精确成立，但我们发现从这种方法得出的结果与实际步长梯度下降方法得到的经验结果有极好的一致性。
URL
https://arxiv.org/abs/2509.23887
