Abstract
Quantized neural network training optimizes a discrete, non-differentiable objective. The straight-through estimator (STE) enables backpropagation through surrogate gradients and is widely used. While previous studies have primarily focused on the properties of surrogate gradients and their convergence, the influence of quantization hyperparameters, such as bit width and quantization range, on learning dynamics remains largely unexplored. We theoretically show that in the high-dimensional limit, STE dynamics converge to a deterministic ordinary differential equation. This reveals that STE training exhibits a plateau followed by a sharp drop in generalization error, with plateau length depending on the quantization range. A fixed-point analysis quantifies the asymptotic deviation from the unquantized linear model. We also extend analytical techniques for stochastic gradient descent to nonlinear transformations of weights and inputs.
Abstract (translated)
量化神经网络训练优化的是一个离散且非可微的目标函数。直通估计器（STE）通过替代梯度使得反向传播成为可能，并被广泛使用。尽管先前的研究主要集中在替代梯度的性质及其收敛性上，但量化超参数（如位宽和量化范围）对学习动态的影响仍然很大程度上未被探索。我们从理论上证明，在高维极限下，STE动力学会收敛到一个确定性的常微分方程。这揭示了STE训练在泛化误差上会出现一个平台期后紧随其后的急剧下降的现象，并且这个平台期的长度取决于量化范围。通过固定点分析可以量化与未量化的线性模型的渐近偏离程度。我们还扩展了随机梯度下降的分析技术，以适用于权重和输入的非线性变换。
URL
https://arxiv.org/abs/2510.10693
