Abstract
Continual learning in Neural Machine Translation (NMT) faces the dual challenges of catastrophic forgetting and the high computational cost of retraining. This study establishes Low-Rank Adaptation (LoRA) as a parameter-efficient framework to address these challenges in dedicated NMT architectures. We first demonstrate that LoRA-based fine-tuning adapts NMT models to new languages and domains with performance on par with full-parameter techniques, while utilizing only a fraction of the parameter space. Second, we propose an interactive adaptation method using a calibrated linear combination of LoRA modules. This approach functions as a gate-free mixture of experts, enabling real-time, user-controllable adjustments to domain and style without retraining. Finally, to mitigate catastrophic forgetting, we introduce a novel gradient-based regularization strategy specifically designed for low-rank decomposition matrices. Unlike methods that regularize the full parameter set, our approach weights the penalty on the low-rank updates using historical gradient information. Experimental results indicate that this strategy efficiently preserves prior domain knowledge while facilitating the acquisition of new tasks, offering a scalable paradigm for interactive and continual NMT.
Abstract (translated)
在神经机器翻译（NMT）中的持续学习面临灾难性遗忘和重新训练高计算成本的双重挑战。本研究建立了低秩适应（LoRA）作为解决这些挑战的有效参数框架，特别是在专门针对NMT架构的情况下。 首先，我们展示了基于LoRA的微调可以使NMT模型适应新的语言和领域，并且其性能与全参数技术相当，但仅使用了一小部分参数空间。其次，我们提出了一种交互式适应方法，该方法采用校准后的线性组合来融合LoRA模块，这可以作为无门控专家混合体工作，能够在不重新训练的情况下实时调整领域和风格。 最后，为了解决灾难性遗忘问题，我们引入了一种新的基于梯度的正则化策略，专门针对低秩分解矩阵。与对整个参数集进行规则化的常规方法不同，我们的方法利用历史梯度信息来加权低秩更新的惩罚。实验结果表明，这种策略能够有效地保留先前领域的知识，同时促进新任务的学习，为交互式和持续的NMT提供了一种可扩展的方法论。 简而言之，本研究通过LoRA及其创新性正则化方法，在减少计算负担的同时提高了模型在多语言、多领域设置下的适应性和持久学习能力。
URL
https://arxiv.org/abs/2512.09910
