Abstract
In the era of large-scale neural network models, optimization algorithms often struggle with generalization due to an overreliance on training loss. One key insight widely accepted in the machine learning community is the idea that wide basins (regions around a local minimum where the loss increases gradually) promote better generalization by offering greater stability to small changes in input data or model parameters. In contrast, sharp minima are typically more sensitive and less stable. Motivated by two key empirical observations - the inherent heavy-tailed distribution of gradient noise in stochastic gradient descent and the Edge of Stability phenomenon during neural network training, in which curvature grows before settling at a plateau, we introduce Adaptive Heavy Tailed Stochastic Gradient Descent (AHTSGD). The algorithm injects heavier-tailed noise into the optimizer during the early stages of training to enhance exploration and gradually transitions to lighter-tailed noise as sharpness stabilizes. By dynamically adapting to the sharpness of the loss landscape throughout training, AHTSGD promotes accelerated convergence to wide basins. AHTSGD is the first algorithm to adjust the nature of injected noise into an optimizer based on the Edge of Stability phenomenon. AHTSGD consistently outperforms SGD and other noise-based methods on benchmarks like MNIST and CIFAR-10, with marked gains on noisy datasets such as SVHN. It ultimately accelerates early training from poor initializations and improves generalization across clean and noisy settings, remaining robust to learning rate choices.
Abstract (translated)
在大规模神经网络模型的时代，优化算法由于过度依赖训练损失而常常难以实现泛化。机器学习社区广泛接受的一个关键见解是：宽盆地（即围绕局部最小值的区域，在该区域内损失逐渐增加）通过提供对输入数据或模型参数的小变化更稳定的特性来促进更好的泛化能力；相比之下，尖锐极小值通常更加敏感且不够稳定。 受到两个重要经验观察结果的启发——随机梯度下降中的梯度噪声呈现固有的重尾分布以及在神经网络训练过程中边缘稳定性现象的存在，在这一现象中曲率会在达到平台期之前先增长。我们提出了自适应重尾随机梯度下降（AHTSGD）。该算法在训练初期向优化器注入更重尾的噪声以增强探索，并随着尖锐性的稳定逐渐过渡到较轻尾的噪声。通过在整个训练过程中动态地调整注入噪声的性质，AHTSGD促进了加速收敛于宽盆地。 AHTSGD是首个根据边缘稳定性现象来调整输入给优化器中的噪声特性的算法。在MNIST和CIFAR-10等基准测试中，AHTSGD始终优于随机梯度下降（SGD）和其他基于噪声的方法，在SVHN这样的嘈杂数据集上表现出显著的改进效果。它最终加快了从较差初始化开始时的早期训练速度，并在整个清洁与嘈杂环境设置下提高了泛化能力，同时对学习率的选择保持鲁棒性。
URL
https://arxiv.org/abs/2508.21353
