Abstract
Continual learning of pretrained vision-language models (VLMs) is prone to catastrophic forgetting, yet current approaches adapt to new tasks without explicitly preserving the cross-modal semantic geometry inherited from pretraining and previous stages, allowing new-task supervision to induce geometric distortion. We observe that the most pronounced drift tends to concentrate in vulnerable neighborhoods near the old-new semantic interface, where shared visual patterns are easily re-explained by new textual semantics. To address this under an exemplar-free constraint, we propose Semantic Geometry Preservation for Continual Learning (SeGP-CL). SeGP-CL first probes the drift-prone region by constructing a compact set of adversarial anchors with dual-targeted projected gradient descent (DPGD), which drives selected new-task seeds toward old-class semantics while remaining faithful in raw visual space. During training, we preserve cross-modal structure by anchor-guided cross-modal geometry distillation (ACGD), and stabilize the textual reference frame across tasks via a lightweight text semantic-geometry regularization (TSGR). After training, we estimate anchor-induced raw-space drift to transfer old visual prototypes and perform dual-path inference by fusing cross-modal and visual cues. Extensive experiments on five continual learning benchmarks demonstrate that SeGP-CL consistently improves stability and forward transfer, achieving state-of-the-art performance while better preserving semantic geometry of VLMs.
Abstract (translated)
持续学习预训练的视觉语言模型（VLM）容易出现灾难性遗忘，而当前的方法在适应新任务时没有显式地保留从预训练和先前阶段继承来的跨模态语义几何结构，这使得新的监督可能导致几何畸变。我们观察到最明显的漂移往往集中在旧任务与新任务之间的脆弱邻域中，在这些区域共享的视觉模式很容易被新的文本语义重新解释。 为了在没有示例的情况下解决这个问题，我们提出了用于持续学习的语义几何保持（Semantic Geometry Preservation for Continual Learning, SeGP-CL）。SeGP-CL 首先通过使用双重目标投影梯度下降法 (DPGD) 构建一组紧凑的对抗性锚点来探测易漂移区域。这种方法引导新任务种子向着旧类语义移动，同时保持在原始视觉空间中的忠实度。 在训练过程中，我们利用锚点指导下的跨模态几何知识蒸馏（ACGD）保留了跨模态结构，并通过轻量级的文本语义几何正则化（TSGR）在整个任务中稳定文本参考框架。经过训练后，我们估计由锚点引起的原始空间漂移来传输旧视觉原型，并进行双路径推理以融合跨模态和视觉线索。 在五个持续学习基准上的广泛实验表明，SeGP-CL 一致地提高了稳定性和前向迁移能力，在保持 VLM 的语义几何结构的同时实现了最先进的性能。
URL
https://arxiv.org/abs/2603.12055
