Abstract
Deep neural networks have achieved remarkable accomplishments in practice. The success of these networks hinges on effective initialization methods, which are vital for ensuring stable and rapid convergence during training. Recently, initialization methods that maintain identity transition within layers have shown good efficiency in network training. These techniques (e.g., Fixup) set specific weights to zero to achieve identity control. However, settings of remaining weight (e.g., Fixup uses random values to initialize non-zero weights) will affect the inductive bias that is achieved only by a zero weight, which may be harmful to training. Addressing this concern, we introduce fully identical initialization (IDInit), a novel method that preserves identity in both the main and sub-stem layers of residual networks. IDInit employs a padded identity-like matrix to overcome rank constraints in non-square weight matrices. Furthermore, we show the convergence problem of an identity matrix can be solved by stochastic gradient descent. Additionally, we enhance the universality of IDInit by processing higher-order weights and addressing dead neuron problems. IDInit is a straightforward yet effective initialization method, with improved convergence, stability, and performance across various settings, including large-scale datasets and deep models.
Abstract (translated)
深度神经网络在实践中已经取得了显著的成就。这些网络的成功很大程度上取决于有效的初始化方法，这对于确保训练过程中的稳定和快速收敛至关重要。最近，保持层内恒等变换（identity transition）的初始化方法显示出在网络训练中良好的效率。例如，Fixup技术通过将特定权重设置为零来实现恒等控制。然而，剩余权重的设定（如Fixup使用随机值来初始化非零权重）会影响仅由零权重所达成的归纳偏差，这可能会对训练产生负面影响。 针对这一问题，我们引入了一种新的方法——完全相同的初始化（IDInit），该方法在残差网络的主要和子茎层中都保持恒等变换。IDInit采用填充后的类似恒等矩阵来克服非方阵权重矩阵中的秩约束。此外，我们还展示了如何通过随机梯度下降法解决恒等矩阵的收敛问题。为了增强IDInit的通用性，我们处理了高阶权重，并解决了“死神经元”（dead neuron）的问题。 总之，IDInit是一种简单而有效的方法，在各种设置下提高了收敛性、稳定性和性能，包括大规模数据集和深度模型。
URL
https://arxiv.org/abs/2503.04626
