Abstract
Balancing sensitivity to new tasks and stability for retaining past knowledge is crucial in continual learning (CL). Recently, sharpness-aware minimization has proven effective in transfer learning and has also been adopted in continual learning (CL) to improve memory retention and learning efficiency. However, relying on zeroth-order sharpness alone may favor sharper minima over flatter ones in certain settings, leading to less robust and potentially suboptimal solutions. In this paper, we propose \textbf{C}ontinual \textbf{Flat}ness (\textbf{C-Flat}), a method that promotes flatter loss landscapes tailored for CL. C-Flat offers plug-and-play compatibility, enabling easy integration with minimal modifications to the code pipeline. Besides, we present a general framework that integrates C-Flat into all major CL paradigms and conduct comprehensive comparisons with loss-minima optimizers and flat-minima-based CL methods. Our results show that C-Flat consistently improves performance across a wide range of settings. In addition, we introduce C-Flat++, an efficient yet effective framework that leverages selective flatness-driven promotion, significantly reducing the update cost required by C-Flat. Extensive experiments across multiple CL methods, datasets, and scenarios demonstrate the effectiveness and efficiency of our proposed approaches. Code is available at this https URL.
Abstract (translated)
在连续学习（CL）中，平衡对新任务的敏感性和保持过去知识的稳定性至关重要。最近，敏锐度感知最小化在迁移学习中被证明是有效的，并已被用于连续学习以改善记忆保留和学习效率。然而，在某些情况下，仅依赖零阶敏锐度可能会倾向于选择更尖锐而非平坦的极小值，从而导致解决方案不够稳健且可能次优。为此，本文提出了**Continual Flatness (C-Flat)** 方法，该方法旨在促进适合连续学习的更加平缓的损失景观。C-Flat 具有即插即用兼容性，可轻松集成到代码流程中而无需大量修改。 此外，我们还提出了一种通用框架，将 C-Flat 整合到所有主要的 CL 方法中，并进行了全面比较，对比了基于损失极小值优化器和基于平坦极小值的 CL 方法。我们的结果显示，在各种设置下，C-Flat 一致性地提高了性能表现。 此外，我们还引入了 C-Flat++，这是一种高效且有效的框架，利用选择性平滑驱动提升，显著减少了由 C-Flat 所需的更新成本。在多个 CL 方法、数据集和场景上的广泛实验展示了我们的方法的有效性和效率。代码可在提供的链接中获取。
URL
https://arxiv.org/abs/2508.18860
