Abstract
Online Continual Learning (OCL) for image classification represents a challenging subset of Continual Learning, focusing on classifying images from a stream without assuming data independence and identical distribution (i.i.d). The primary challenge in this context is to prevent catastrophic forgetting, where the model's performance on previous tasks deteriorates as it learns new ones. Although various strategies have been proposed to address this issue, achieving rapid convergence remains a significant challenge in the online setting. In this work, we introduce a novel approach to training OCL models that utilizes the Natural Gradient Descent optimizer, incorporating an approximation of the Fisher Information Matrix (FIM) through Kronecker Factored Approximate Curvature (KFAC). This method demonstrates substantial improvements in performance across all OCL methods, particularly when combined with existing OCL tricks, on datasets such as Split CIFAR-100, CORE50, and Split miniImageNet.
Abstract (translated)
在线持续学习（OCL）图像分类是持续学习中的一个具有挑战性的子领域，其核心在于处理来自数据流的图像分类任务，且不假设数据满足独立同分布（i.i.d.）。该场景下的主要挑战是防止灾难性遗忘——即模型在学习新任务时对先前任务的性能会逐渐退化。尽管已有多种策略被提出以解决此问题，但在在线学习场景中实现快速收敛仍是一项重大挑战。本研究提出了一种训练OCL模型的新方法，该方法采用自然梯度下降（Natural Gradient Descent）优化器，并通过克罗内克因子近似曲率（KFAC）对费雪信息矩阵（FIM）进行近似。实验表明，该方法在Split CIFAR-100、CORE50和Split miniImageNet等数据集上，能够显著提升所有OCL方法的性能，尤其在与现有OCL技巧结合时效果更为突出。
URL
https://arxiv.org/abs/2603.20898
