Abstract
Training large neural networks through gradient-based optimization requires navigating high-dimensional loss landscapes, which often exhibit pathological geometry, leading to undesirable training dynamics. In particular, poor generalization frequently results from convergence to sharp minima that are highly sensitive to input perturbations, causing the model to overfit the training data while failing to generalize to unseen examples. Furthermore, these optimization procedures typically display strong dependence on the fine structure of the loss landscape, leading to unstable training dynamics, due to the fractal-like nature of the loss surface. In this work, we propose an alternative optimizer that simultaneously reduces this dependence, and avoids sharp minima, thereby improving generalization. This is achieved by simulating the motion of the center of a ball rolling on the loss landscape. The degree to which our optimizer departs from the standard gradient descent is controlled by a hyperparameter, representing the radius of the ball. Changing this hyperparameter allows for probing the loss landscape at different scales, making it a valuable tool for understanding its geometry.
Abstract (translated)
通过基于梯度的优化训练大型神经网络需要在高维损失景观中导航，而这种景观通常具有病态几何形状，导致不理想的训练动态。特别是，由于模型收敛于对输入扰动高度敏感的尖锐极小值，从而过拟合训练数据并无法泛化到未见示例，因此经常导致较差的一般性（泛化能力）。此外，这些优化过程通常强烈依赖于损失景观的精细结构，导致不稳定的训练动态，这是由于损失表面具有分形特性。在这项工作中，我们提出了一种替代的优化器，该优化器同时减少了这种依赖关系并避免了尖锐极小值，从而提高了泛化能力。这通过模拟一个球心在损失景观上滚动的运动来实现。我们的优化器与标准梯度下降方法的区别程度由超参数表示（即球体的半径）控制。改变这个超参数可以在不同尺度下探测损失景观，使其成为了解其几何形状的重要工具。
URL
https://arxiv.org/abs/2505.19527
