Abstract
Adam is widely recognized as one of the most effective optimizers for training deep neural networks (DNNs). Despite its remarkable empirical success, its theoretical convergence analysis remains unsatisfactory. Existing works predominantly interpret Adam as a preconditioned stochastic gradient descent with momentum (SGDM), formulated as $\bm{x}_{t+1} = \bm{x}_t - \frac{\gamma_t}{{\sqrt{\bm{v}_t}+\epsilon}} \circ \bm{m}_t$. This perspective necessitates strong assumptions and intricate techniques, resulting in lengthy and opaque convergence proofs that are difficult to verify and extend. In contrast, we propose a novel interpretation by treating Adam as a sign-like optimizer, expressed as $\bm{x}_{t+1} = \bm{x}_t - \gamma_t \frac{|\bm{m}_t|}{{\sqrt{\bm{v}_t}+\epsilon}} \circ {\rm Sign}(\bm{m}_t)$. This reformulation significantly simplifies the convergence analysis. For the first time, with some mild conditions, we prove that Adam achieves the optimal rate of ${\cal O}(\frac{1}{T^{\sfrac{1}{4}}})$ rather than the previous ${\cal O} \left(\frac{\ln T}{T^{\sfrac{1}{4}}}\right)$ under weak assumptions of the generalized $p$-affine variance and $(L_0, L_1, q)$-smoothness, without dependence on the model dimensionality or the numerical stability parameter $\epsilon$. Additionally, our theoretical analysis provides new insights into the role of momentum as a key factor ensuring convergence and offers practical guidelines for tuning learning rates in Adam, further bridging the gap between theory and practice.
Abstract (translated)
亚当（Adam）优化算法被广泛认为是训练深度神经网络（DNN）中最有效的优化器之一。尽管其实验结果令人印象深刻，但其理论收敛性分析仍然不够完善。现有研究主要将Adam解释为带有动量的预处理随机梯度下降（SGDM），公式化表达为 $\bm{x}_{t+1} = \bm{x}_t - \frac{\gamma_t}{\sqrt{\bm{v}_t}+\epsilon} \circ \bm{m}_t$。这种视角需要强假设和复杂的技术，导致收敛性证明冗长且难以验证和扩展。相比之下，我们提出了一种新的解释方法，即将Adam视为类似符号的优化器（sign-like optimizer），公式化表达为 $\bm{x}_{t+1} = \bm{x}_t - \gamma_t \frac{|\bm{m}_t|}{\sqrt{\bm{v}_t}+\epsilon} \circ {\rm Sign}(\bm{m}_t)$。这种重新表述显著简化了收敛性分析。 我们首次证明，在广义$p$仿射方差和$(L_0, L_1, q)$平滑性的弱假设下，且在不依赖于模型维度或数值稳定性参数$\epsilon$的情况下，Adam的收敛速度为${\cal O}(\frac{1}{T^{\sfrac{1}{4}}})$，而不是之前的${\cal O} \left(\frac{\ln T}{T^{\sfrac{1}{4}}}\right)$。此外，我们的理论分析还提供了关于动量作为确保收敛的关键因素的新见解，并为调整Adam中的学习率提供实用指导，进一步缩小了理论与实践之间的差距。
URL
https://arxiv.org/abs/2507.05966
