Abstract
Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. The resulting update direction is invariant under reparameterization, guarantees descent in the Fisher metric, and helps preserve prior task outputs. We provide theoretical analysis establishing the properties of the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100.
Abstract (translated)
持续学习旨在使神经网络能够在顺序任务中获取新知识。然而，在这种设置下的关键挑战是学习新任务的同时不灾难性地忘记以前学过的任务。我们提出了Fisher-正交投影自然梯度下降（FOPNG）优化器，它在参数更新时施加了Fisher-正交约束，以保持旧任务的性能同时学习新任务。与现有方法在欧几里得参数空间中操作不同，FOPNG将梯度投影到以前任务梯度的Fisher-正交补集中。这种方法统一了自然梯度下降和正交梯度方法在一个信息几何框架内。由此产生的更新方向对于重参数化是不变的，在Fisher度量下保证了下降，并有助于保持先前任务的输出。 我们提供了关于投影更新性质的理论分析，描述了使用对角Fisher的有效且实用实现，并在Permuted-MNIST、Split-MNIST、Rotated-MNIST、Split-CIFAR10和Split-CIFAR100等标准持续学习基准测试中展示了强大的结果。
URL
https://arxiv.org/abs/2601.12816
