Abstract
Euclidean gradient descent algorithms barely capture the geometry of objective function-induced hypersurfaces and risk driving update trajectories off the hypersurfaces. Riemannian gradient descent algorithms address these issues but fail to represent complex hypersurfaces via a single classic manifold. We propose geodesic gradient descent (GGD), a generic and learning-rate-free Riemannian gradient descent algorithm. At each iteration, GGD uses an n-dimensional sphere to approximate a local neighborhood on the objective function-induced hypersurface, adapting to arbitrarily complex geometries. A tangent vector derived from the Euclidean gradient is projected onto the sphere to form a geodesic, ensuring the update trajectory stays on the hypersurface. Parameter updates are performed using the endpoint of the geodesic. The maximum step size of the gradient in GGD is equal to a quarter of the arc length on the n-dimensional sphere, thus eliminating the need for a learning rate. Experimental results show that compared with the classic Adam algorithm, GGD achieves test MSE reductions ranging from 35.79% to 48.76% for fully connected networks on the Burgers' dataset, and cross-entropy loss reductions ranging from 3.14% to 11.59% for convolutional neural networks on the MNIST dataset.
Abstract (translated)
欧几里得梯度下降算法几乎无法捕捉目标函数生成的超曲面几何结构，并且可能使更新轨迹偏离该超曲面。黎曼梯度下降算法解决了这些问题，但未能通过单一的经典流形表示复杂的超曲面。我们提出了一种通用且无需学习率的黎曼梯度下降算法——测地线梯度下降（Geodesic Gradient Descent, GGD）。在每次迭代中，GGD 使用一个 n 维球体来近似目标函数生成的超曲面上的一个局部邻域，并适应任意复杂的几何结构。通过将由欧几里得梯度导出的切向量投影到该球体上形成测地线，确保更新轨迹保持在超曲面上。参数更新使用的是测地线的端点进行。GGD 中梯度的最大步长等于 n 维球上的四分之一弧长，从而消除了对学习率的需求。 实验结果表明，与经典的 Adam 算法相比，在 Burgers 数据集上用于完全连接网络时，GGD 可实现测试 MSE 的减少范围为 35.79% 至 48.76%，在 MNIST 数据集上的卷积神经网络则可实现交叉熵损失的减少范围为 3.14% 至 11.59%。
URL
https://arxiv.org/abs/2603.06651
