Abstract
Contrary to common belief, we show that gradient ascent-based unconstrained optimization methods frequently fail to perform machine unlearning, a phenomenon we attribute to the inherent statistical dependence between the forget and retain data sets. This dependence, which can manifest itself even as simple correlations, undermines the misconception that these sets can be independently manipulated during unlearning. We provide empirical and theoretical evidence showing these methods often fail precisely due to this overlooked relationship. For random forget sets, this dependence means that degrading forget set metrics (which, for a retrained model, should mirror test set metrics) inevitably harms overall test performance. Going beyond random sets, we consider logistic regression as an instructive example where a critical failure mode emerges: inter-set dependence causes gradient descent-ascent iterations to progressively diverge from the ideal retrained model. Strikingly, these methods can converge to solutions that are not only far from the retrained ideal but are potentially even further from it than the original model itself, rendering the unlearning process actively detrimental. A toy example further illustrates how this dependence can trap models in inferior local minima, inescapable via finetuning. Our findings highlight that the presence of such statistical dependencies, even when manifest only as correlations, can be sufficient for ascent-based unlearning to fail. Our theoretical insights are corroborated by experiments on complex neural networks, demonstrating that these methods do not perform as expected in practice due to this unaddressed statistical interplay.
Abstract (translated)
与普遍看法相反，我们展示了基于梯度上升的无约束优化方法在执行机器遗忘（machine unlearning）时经常失败的现象。我们将这一现象归因于忘记集和保留集之间的固有统计依赖性。这种依赖关系即使表现为简单的相关性，也会破坏这样一种误解，即这些集合可以在不学习过程中独立操作。我们提供了实证和理论证据，表明正是这种被忽视的关系导致了方法的失效。 对于随机选择的遗忘集而言，这种依赖意味着降低忘记集指标（在重新训练模型时应与测试集指标一致）不可避免地会损害整体测试性能。超越随机集合的情况，我们将逻辑回归作为示例来说明一个重要失败模式：跨集依赖性导致梯度下降-上升迭代逐渐偏离理想重新训练后的模型。值得注意的是，这些方法可以收敛到距离重新训练的理想结果很远甚至比原始模型还差的解决方案上，使不学习过程变得有害。 一个玩具例子进一步展示了这种依赖如何将模型困在次优局部最小值中，无法通过微调逃脱。我们的研究发现强调了即使只表现为相关性，这样的统计依赖关系就足以导致基于上升的方法失效。我们理论见解得到了在复杂神经网络上的实验验证，这些实验表明由于未解决的统计交互作用，在实践中这些方法的表现并不如预期。 简而言之，研究表明传统优化方法在执行机器遗忘任务时常常失败，并且这种失败是由数据集间的统计依赖性引起的，这与人们通常认为的数据集可以独立处理的看法相悖。
URL
https://arxiv.org/abs/2509.26427
