Abstract
In this work, we present DEcoupLEd Distillation To Erase (DELETE), a general and strong unlearning method for any class-centric tasks. To derive this, we first propose a theoretical framework to analyze the general form of unlearning loss and decompose it into forgetting and retention terms. Through the theoretical framework, we point out that a class of previous methods could be mainly formulated as a loss that implicitly optimizes the forgetting term while lacking supervision for the retention term, disturbing the distribution of pre-trained model and struggling to adequately preserve knowledge of the remaining classes. To address it, we refine the retention term using "dark knowledge" and propose a mask distillation unlearning method. By applying a mask to separate forgetting logits from retention logits, our approach optimizes both the forgetting and refined retention components simultaneously, retaining knowledge of the remaining classes while ensuring thorough forgetting of the target class. Without access to the remaining data or intervention (i.e., used in some works), we achieve state-of-the-art performance across various benchmarks. What's more, DELETE is a general solution that can be applied to various downstream tasks, including face recognition, backdoor defense, and semantic segmentation with great performance.
Abstract (translated)
在这项工作中，我们提出了DEcoupLEd Distillation To Erase (DELETE)，这是一种适用于任何以类别为中心任务的通用且强大的撤销学习方法。为了导出这种方法，我们首先提出了一种理论框架来分析撤销损失的一般形式，并将其分解为遗忘和保留两个部分。通过该理论框架，我们指出先前一类方法的主要问题在于它们主要优化的是隐式的遗忘项损失，而缺乏对保留项的监督机制，这会扰乱预训练模型的数据分布并难以充分保持剩余类别的知识。 为了解决这个问题，我们利用“暗知识”来改进保留项，并提出了一种掩码蒸馏撤销学习方法。通过应用掩码将遗忘得分从保留得分中分离出来，我们的方法可以同时优化遗忘和改进后的保留部分，在保留其他类别知识的同时确保彻底忘记目标类别的信息。 无需访问剩余数据或干预（即在一些工作中所使用的做法），我们在各种基准测试上达到了最先进的性能水平。更重要的是，DELETE是一种通用解决方案，可应用于包括人脸识别、后门防御以及语义分割在内的多种下游任务，并且具有出色的性能表现。
URL
https://arxiv.org/abs/2503.23751
