Abstract
Gradient descent based optimization methods are the methods of choice to train deep neural networks in machine learning. Beyond the standard gradient descent method, also suitable modified variants of standard gradient descent involving acceleration techniques such as the momentum method and/or adaptivity techniques such as the RMSprop method are frequently considered optimization methods. These days the most popular of such sophisticated optimization schemes is presumably the Adam optimizer that has been proposed in 2014 by Kingma and Ba. A highly relevant topic of research is to investigate the speed of convergence of such optimization methods. In particular, in 1964 Polyak showed that the standard gradient descent method converges in a neighborhood of a strict local minimizer with rate (x - 1)(x + 1)^{-1} while momentum achieves the (optimal) strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} where x \in (1,\infty) is the condition number (the ratio of the largest and the smallest eigenvalue) of the Hessian of the objective function at the local minimizer. It is the key contribution of this work to reveal that Adam also converges with the strictly faster convergence rate (\sqrt{x} - 1)(\sqrt{x} + 1)^{-1} while RMSprop only converges with the convergence rate (x - 1)(x + 1)^{-1}.
Abstract (translated)
基于梯度下降的优化方法是训练深度神经网络在机器学习中的首选方法。除了标准的梯度下降方法外，还包括适合加速技术（如动量法）和适应性技术（如RMSprop方法）的标准梯度下降的各种变体，这些技术也常被考虑作为优化方法。近年来，最流行的此类复杂优化方案可能是Kingma和Ba在2014年提出的Adam优化器。研究这样一个优化方法的收敛速度是一个非常重要的课题。特别地，在1964年Polyak证明了标准梯度下降方法在一个严格局部极小值附近的收敛率为(x - 1)(x + 1)^{-1}，而动量法则实现了最优且更快速的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}，其中x \in (1,\infty)是目标函数在局部极小值点处Hessian矩阵的最大特征值与最小特征值比（即条件数）。这项工作的关键贡献在于揭示了Adam优化器也以严格更快的收敛率(\sqrt{x} - 1)(\sqrt{x} + 1)^{-1}进行收敛，而RMSprop仅以(x - 1)(x + 1)^{-1}的收敛速率进行收敛。
URL
https://arxiv.org/abs/2504.19426
