Abstract
Neural networks often rely on spurious shortcuts for many epochs before discovering structured representations. However, the mechanism governing when this transition occurs and whether its timing can be predicted remains unclear. Prior work shows that gradient descent converges to low norm solutions and that neural networks exhibit simplicity bias, but neither explains the timescale of the transition from shortcut features to structured representations. We introduce the Norm-Hierarchy Transition (NHT) framework, which explains delayed representation learning as the slow traversal of a hierarchy of parameter norms during regularized optimization. When multiple interpolating solutions exist with different norms, weight decay gradually moves the model from high norm shortcut solutions toward lower norm structured representations. We derive a tight bound showing that the transition delay grows logarithmically with the ratio between shortcut and structured norms. Experiments on modular arithmetic, CIFAR-10 with spurious features, CelebA, and Waterbirds support the predictions of the framework. The results suggest that grokking, shortcut learning, and delayed feature discovery arise from a common mechanism based on norm hierarchy traversal during training.
Abstract (translated)
神经网络常常在发现结构化表示之前依赖于伪捷径（即无关紧要的特征）很多个周期。然而，决定这一转换何时发生以及其时间是否可以预测的机制仍然不清楚。先前的工作表明，梯度下降收敛到低范数解，并且神经网络表现出简洁性偏见，但这两者都不能解释从伪特征到结构化表示的时间尺度上的过渡过程。 我们引入了基于规范层级转换（Norm-Hierarchy Transition, NHT）框架，该框架将延迟的表示学习解释为在正则化的优化过程中缓慢遍历参数范数层级的过程。当存在多个具有不同范数的插值解时，权重衰减会逐渐地将模型从高范数的伪特征解决方案引导至低范数的结构化表征。 我们推导出一个紧致边界，表明过渡延迟随捷径和结构化范数之间的比率呈对数增长。在模运算、具有伪特征的CIFAR-10、CelebA以及Waterbirds数据集上的实验均验证了该框架的预测结果。 这些研究结果暗示了“理解”（grokking）、捷径学习及延迟特征发现现象源自训练过程中基于规范层级遍历的共同机制。
URL
https://arxiv.org/abs/2603.07323
