Abstract
Orthogonal gradient descent has emerged as a powerful method for continual learning tasks. However, its Euclidean projections overlook the underlying information-geometric structure of the space of distributions parametrized by neural networks, which can lead to suboptimal convergence in learning tasks. To counteract this, we combine it with the idea of the natural gradient and present ONG (Orthogonal Natural Gradient Descent). ONG preconditions each new task gradient with an efficient EKFAC approximation of the inverse Fisher information matrix, yielding updates that follow the steepest descent direction under a Riemannian metric. To preserve performance on previously learned tasks, ONG projects these natural gradients onto the orthogonal complement of prior task gradients. We provide a theoretical justification for this procedure, introduce the ONG algorithm, and benchmark its performance on the Permuted and Rotated MNIST datasets. All code for our experiments/reproducibility can be found at this https URL.
Abstract (translated)
正交梯度下降（Orthogonal gradient descent）已成为连续学习任务中的一种强大方法。然而，其欧几里得投影忽视了由神经网络参数化分布空间所具有的底层信息几何结构，这可能导致在学习任务中的次优收敛。为解决这一问题，我们将正交梯度与自然梯度的概念相结合，并提出了ONG（Orthogonal Natural Gradient Descent）。ONG通过高效地使用EKFAC近似逆费雪信息矩阵来预处理每个新任务的梯度，从而生成遵循黎曼度量下最陡下降方向的更新。为了保持在之前学习的任务上的性能，ONG将这些自然梯度投影到先前任务梯度正交补集上。我们为这一过程提供了理论依据，并引入了ONG算法，在Permuted和Rotated MNIST数据集上对其进行了基准测试以展示其效果。 所有实验代码及重现性所需的内容都可以在以下网址找到：[此处插入具体URL]。
URL
https://arxiv.org/abs/2508.17169
