Abstract
Our theoretical understanding of neural networks is lagging behind their empirical success. One of the important unexplained phenomena is why and how, during the process of training with gradient descent, the theoretical capacity of neural networks is reduced to an effective capacity that fits the task. We here investigate the mechanism by which gradient descent achieves this through analyzing the learning dynamics at the level of individual neurons in single hidden layer ReLU networks. We identify three dynamical principles -- mutual alignment, unlocking and racing -- that together explain why we can often successfully reduce capacity after training through the merging of equivalent neurons or the pruning of low norm weights. We specifically explain the mechanism behind the lottery ticket conjecture, or why the specific, beneficial initial conditions of some neurons lead them to obtain higher weight norms.
Abstract (translated)
我们对神经网络的理论理解滞后于它们的经验成功。一个重要的未解释现象是，在使用梯度下降法进行训练的过程中，为什么和如何，神经网络的理论容量会被减少到一种有效容量以适应任务需求。本文通过分析单隐藏层ReLU网络中个别神经元的学习动力学来探讨梯度下降是如何实现这一过程的机制。我们识别出三个动态原则——互对齐、解锁和竞速，这些共同解释了为什么在训练后往往可以通过等效神经元合并或低权重裁剪成功地减少容量。具体来说，本文解释了彩票票证猜想背后的机制，即为何一些特定初始条件有利的神经元会获得更高的权重范数。
URL
https://arxiv.org/abs/2602.04832
