Abstract
Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new research into low-rank training, compression, and adaptation, theoretical justification for these dynamics in nonlinear networks remains limited. %compared to deep linear settings. To address this gap, this paper analyzes the learning dynamics of multi-layer perceptrons (MLPs) under gradient descent (GD). We demonstrate that the weight dynamics concentrate within invariant low-dimensional subspaces throughout training. Theoretically, we precisely characterize these invariant subspaces for two-layer networks with smooth nonlinear activations, providing insight into their emergence. Experimentally, we validate that this phenomenon extends beyond our theoretical assumptions. Leveraging these insights, we empirically show there exists a low-rank MLP parameterization that, when initialized within the appropriate subspaces, matches the classification performance of fully-parameterized counterparts on a variety of classification tasks.
Abstract (translated)
最近的实证证据表明，大规模深度神经网络的训练动力学发生在低维子空间中。尽管这激发了对低秩训练、压缩和适应性的新研究，但非线性网络中这些动态行为的理论依据仍然有限。 为填补这一空白，本文分析了在梯度下降（GD）下多层感知器（MLPs）的学习动力学。我们展示了权重的动力学在整个训练过程中集中在不变的低维子空间内。理论上，对于具有平滑非线性激活的两层网络，我们精确地描述了这些不变子空间，为它们的出现提供了见解。实验上，我们验证了这一现象超越了我们的理论假设。 借助这些见解，我们在经验上展示了存在一个低秩MLP参数化方式，在适当初始化后，其在各种分类任务上的分类性能与完全参数化的同类网络相当。
URL
https://arxiv.org/abs/2602.06208
