Abstract
Recent studies have proposed interpreting the training process from an ergodic perspective. Building on this foundation we present a unified framework for understanding and accelerating the training of deep neural networks via stochastic gradient descent. By analyzing the geometric landscape of the objective function we introduce a practical diagnostic, the running estimate of the largest Lyapunov exponent, which provably distinguishes genuine convergence toward stable minimizers from mere statistical stabilization near saddle points. We then propose a ghost category extension for standard classifiers that adds auxiliary ghost output nodes so the model gains extra descent directions that open a lateral corridor around narrow loss barriers and enable the optimizer to bypass poor basins during the early training phase. We show that this extension strictly reduces approximation error and that after sufficient convergence the ghost dimensions collapse and the extended model's invariant law coincides with that of the original and there exists a path in the enlarged parameter space along which the total loss does not increase while the original loss decreases by an arbitrary margin. Taken together these results provide a principled architecture level intervention that accelerates early stage trainability while preserving asymptotic behavior.
Abstract (translated)
近期的研究提出从遍历性视角来解释训练过程。在此基础上，我们提出了一个统一的框架，通过随机梯度下降法理解并加速深度神经网络的训练。通过对目标函数几何景观的分析，我们引入了一个实用的诊断工具——运行中的最大Lyapunov指数估计值，该方法可以证明区分真正的向稳定极小值点收敛和仅仅在鞍点附近的统计稳定。 接着，我们提出了一种标准分类器的幽灵类别扩展方法，即添加辅助的幽灵输出节点。这样模型可以获得额外的下降方向，在狭窄损失障碍物周围打开横向通道，并使优化器能够在训练初期绕过不良盆地。 我们展示了这种扩展能严格减少近似误差，并且在充分收敛后，幽灵维度会坍缩，扩展后的模型不变性定律与原模型一致，同时在扩大的参数空间中存在一条路径，在这条路径上总损失不会增加而原始损失可以任意减小。 综上所述，这些结果提供了一种基于架构的干预方法，可以在保持渐近行为的同时加速早期训练阶段。
URL
https://arxiv.org/abs/2507.01003
