Abstract
Lipschitz continuity characterizes the worst-case sensitivity of neural networks to small input perturbations; yet its dynamics (i.e. temporal evolution) during training remains under-explored. We present a rigorous mathematical framework to model the temporal evolution of Lipschitz continuity during training with stochastic gradient descent (SGD). This framework leverages a system of stochastic differential equations (SDEs) to capture both deterministic and stochastic forces. Our theoretical analysis identifies three principal factors driving the evolution: (i) the projection of gradient flows, induced by the optimization dynamics, onto the operator-norm Jacobian of parameter matrices; (ii) the projection of gradient noise, arising from the randomness in mini-batch sampling, onto the operator-norm Jacobian; and (iii) the projection of the gradient noise onto the operator-norm Hessian of parameter matrices. Furthermore, our theoretical framework sheds light on such as how noisy supervision, parameter initialization, batch size, and mini-batch sampling trajectories, among other factors, shape the evolution of the Lipschitz continuity of neural networks. Our experimental results demonstrate strong agreement between the theoretical implications and the observed behaviors.
Abstract (translated)
李普希茨连续性表征了神经网络对小输入扰动的最坏情况敏感度；然而，在使用随机梯度下降（SGD）进行训练时，其动力学特性（即随时间的变化过程）仍然研究不足。我们提出了一种严格的数学框架来模拟在使用SGD训练过程中李普希茨连续性的随时间变化过程。该框架利用随机微分方程（SDEs）系统捕捉确定性和随机力的影响。 我们的理论分析识别了驱动演变的三个主要因素： (i) 由优化动力学引起的梯度流向参数矩阵操作员范数雅可比矩阵的投影； (ii) 由于小批量采样中的随机性产生的梯度噪声对操作员范数雅可比矩阵的投影； (iii) 梯度噪声向参数矩阵操作员范数海森矩阵的投影。 此外，我们的理论框架还揭示了诸如嘈杂监督、参数初始化、批次大小和小批量采样轨迹等因素如何塑造神经网络李普希茨连续性随时间的变化。实验结果表明，理论推断与观察到的行为之间有很强的一致性。
URL
https://arxiv.org/abs/2506.18588
