Abstract
Grokking refers to a delayed generalization following overfitting when optimizing artificial neural networks with gradient-based methods. In this work, we demonstrate that grokking can be induced by regularization, either explicit or implicit. More precisely, we show that when there exists a model with a property $P$ (e.g., sparse or low-rank weights) that generalizes on the problem of interest, gradient descent with a small but non-zero regularization of $P$ (e.g., $\ell_1$ or nuclear norm regularization) results in grokking. This extends previous work showing that small non-zero weight decay induces grokking. Moreover, our analysis shows that over-parameterization by adding depth makes it possible to grok or ungrok without explicitly using regularization, which is impossible in shallow cases. We further show that the $\ell_2$ norm is not a reliable proxy for generalization when the model is regularized toward a different property $P$, as the $\ell_2$ norm grows in many cases where no weight decay is used, but the model generalizes anyway. We also show that grokking can be amplified solely through data selection, with any other hyperparameter fixed.
Abstract (translated)
"Grokking"（灵悟）是指在使用基于梯度的方法优化人工神经网络时，在过度拟合之后出现的延迟泛化现象。在这项工作中，我们展示了通过正则化可以诱导出灵悟现象，这种正则化既可以是显式的也可以是隐式的。更具体地说，当存在一个具有某种性质P（例如稀疏或低秩权重）且在感兴趣的问题上能够泛化的模型时，使用带有小但非零的针对该性质P的梯度下降和正则化（例如L1或者核范数正则化），会导致灵悟现象的发生。这扩展了之前的工作，这些工作表明即使是很小的非零权重衰减也能导致灵悟。 此外，我们的分析还显示，通过增加深度进行过度参数化使得模型能够在不显式使用正则化的情况下实现或避免灵悟，这是在浅层网络中不可能做到的。我们进一步展示了当模型被调整以向不同的性质P（例如稀疏性）泛化时，L2范数不能可靠地作为泛化的代理指标：因为在没有权重衰减的情况下，L2范数会在许多情况下增长，但模型仍然可以实现泛化。 此外，我们还表明可以通过单纯的数据选择来放大灵悟现象，而不改变其他的超参数。
URL
https://arxiv.org/abs/2506.05718
