Abstract
Continual Text Classification (CTC) aims to continuously classify new text data over time while minimizing catastrophic forgetting of previously acquired knowledge. However, existing methods often focus on task-specific knowledge, overlooking the importance of shared, task-agnostic knowledge. Inspired by the complementary learning systems theory, which posits that humans learn continually through the interaction of two systems -- the hippocampus, responsible for forming distinct representations of specific experiences, and the neocortex, which extracts more general and transferable representations from past experiences -- we introduce Information-Theoretic Complementary Prompts (InfoComp), a novel approach for CTC. InfoComp explicitly learns two distinct prompt spaces: P(rivate)-Prompt and S(hared)-Prompt. These respectively encode task-specific and task-invariant knowledge, enabling models to sequentially learn classification tasks without relying on data replay. To promote more informative prompt learning, InfoComp uses an information-theoretic framework that maximizes mutual information between different parameters (or encoded representations). Within this framework, we design two novel loss functions: (1) to strengthen the accumulation of task-specific knowledge in P-Prompt, effectively mitigating catastrophic forgetting, and (2) to enhance the retention of task-invariant knowledge in S-Prompt, improving forward knowledge transfer. Extensive experiments on diverse CTC benchmarks show that our approach outperforms previous state-of-the-art methods.
Abstract (translated)
连续文本分类（CTC）的目标是在时间的推移中持续对新文本数据进行分类，同时尽量减少之前获得的知识的灾难性遗忘。然而，现有的方法往往专注于特定任务的知识，而忽视了共享、无任务特异性知识的重要性。受互补学习系统理论的启发——该理论认为人类通过两个系统的相互作用来实现连续学习：海马体负责形成特定经历的独特表示，新皮层则从过去的经验中提取更为普遍和可转移的表示——我们提出了信息论互补提示（InfoComp），这是一种新颖的CTC方法。InfoComp明确地学习两种不同的提示空间：P(rivate)-Prompt（私有提示）和S(hared)-Prompt（共享提示）。它们分别编码特定任务的知识和无任务特异性的知识，使模型能够在不依赖数据重播的情况下逐步学习分类任务。 为了促进更具信息量的提示学习，InfoComp采用了一个基于信息论的框架，该框架最大化了不同参数或表示之间的互信息。在此框架内，我们设计了两个新的损失函数：（1）用于增强P-Prompt中特定任务知识的积累，有效减轻灾难性遗忘；（2）用于提高S-Prompt中无任务特异性知识的保留度，从而改进向前的知识传递。 在多种CTC基准上的广泛实验表明，我们的方法优于先前最先进的方法。
URL
https://arxiv.org/abs/2505.20933
