Abstract
We study the implicit bias of Sharpness-Aware Minimization (SAM) when training $L$-layer linear diagonal networks on linearly separable binary classification. For linear models ($L=1$), both $\ell_\infty$- and $\ell_2$-SAM recover the $\ell_2$ max-margin classifier, matching gradient descent (GD). However, for depth $L = 2$, the behavior changes drastically -- even on a single-example dataset. For $\ell_\infty$-SAM, the limit direction depends critically on initialization and can converge to $\mathbf{0}$ or to any standard basis vector, in stark contrast to GD, whose limit aligns with the basis vector of the dominant data coordinate. For $\ell_2$-SAM, we show that although its limit direction matches the $\ell_1$ max-margin solution as in the case of GD, its finite-time dynamics exhibit a phenomenon we call "sequential feature amplification", in which the predictor initially relies on minor coordinates and gradually shifts to larger ones as training proceeds or initialization increases. Our theoretical analysis attributes this phenomenon to $\ell_2$-SAM's gradient normalization factor applied in its perturbation, which amplifies minor coordinates early and allows major ones to dominate later, giving a concrete example where infinite-time implicit-bias analyses are insufficient. Synthetic and real-data experiments corroborate our findings.
Abstract (translated)
我们在研究训练$L$层线性对角网络（用于线性可分二元分类）时，Sharpness-Aware Minimization (SAM) 的隐式偏差。对于线性模型($L=1$)，不论是$\ell_\infty$-SAM 还是 $\ell_2$-SAM 都恢复了 $\ell_2$ 最大间隔分类器，这与梯度下降（GD）一致。然而，当深度$L = 2$时，行为有了巨大的变化——甚至在单一示例数据集上也是如此。 对于$\ell_\infty$-SAM，极限方向在很大程度上依赖于初始化，并且可以收敛到零向量或任何标准基向量，这与GD形成鲜明对比，GD的极限则与主导数据坐标的基向量对齐。而对于$\ell_2$-SAM，我们证明尽管其极限方向匹配了梯度下降情况下 $\ell_1$ 最大间隔解的情况，但其有限时间动态表现出一种现象我们称之为“顺序特征放大”，即预测器最初依赖于次要坐标，并随着训练的进行或初始化的增加而逐渐转向主要坐标。我们的理论分析认为这种现象归因于$\ell_2$-SAM 在其扰动中应用的梯度标准化因子，这在早期放大了次要坐标并在后期允许主要坐标占据主导地位，从而提供了一个具体例子说明无限时间隐式偏差分析是不够的。 合成数据和实际数据实验验证了我们的发现。
URL
https://arxiv.org/abs/2603.08290
