Abstract
Machine unlearning in large language models (LLMs) is essential for privacy and safety; however, existing approaches remain unstable and unreliable. A widely used strategy, the gradient difference method, applies gradient descent on retained data while performing gradient ascent on forget data, the data whose influence should be removed. However, when combined with cross-entropy loss, this procedure causes unbounded growth of weights and gradients, leading to training instability and degrading both forgetting and retention. We provide a theoretical framework that explains this failure, explicitly showing how ascent on the forget set destabilizes optimization in the feedforward MLP layers of LLMs. Guided by this insight, we propose Bounded Parameter-Efficient Unlearning, a parameter-efficient approach that stabilizes LoRA-based fine-tuning by applying bounded functions to MLP adapters. This simple modification controls the weight dynamics during ascent, enabling the gradient difference method to converge reliably. Across the TOFU, TDEC, and MUSE benchmarks, and across architectures and scales from 125M to 8B parameters, our method achieves substantial improvements in forgetting while preserving retention, establishing a novel theoretically grounded and practically scalable framework for unlearning in LLMs.
Abstract (translated)
大型语言模型（LLMs）中的机器遗忘对于隐私和安全至关重要；然而，现有的方法仍然不稳定且不可靠。广泛使用的一种策略是梯度差分法，在保留的数据上应用梯度下降，同时在应被删除影响的“忘记”数据上执行梯度上升。但是，当与交叉熵损失结合时，这种方法会导致权重和梯度无界增长，从而引起训练不稳定，并降低遗忘和保持的效果。我们提供了一个理论框架来解释这一失败，明确展示了对“忘记集”的上升如何在前馈MLP层中破坏优化的稳定性。 基于这一见解，我们提出了有界的参数高效遗忘（Bounded Parameter-Efficient Unlearning）方法，这是一种参数高效的策略，通过将有界函数应用于LoRA微调中的MLP适配器来稳定基于LoRA的微调。这一简单的修改控制了上升过程中的权重动态，使得梯度差分法能够可靠地收敛。 在TOFU、TDEC和MUSE基准测试中，以及在从125M到80亿参数的不同架构和规模上，我们的方法实现了显著的遗忘改进，并保持了保留的效果。这建立了一个理论基础坚实且实践可扩展的新框架，用于LLMs中的机器遗忘。
URL
https://arxiv.org/abs/2509.24166
