Abstract
Low-precision training has become essential for reducing the computational and memory costs of large-scale deep learning. However, quantization of gradients introduces both magnitude shrinkage and additive noise, which can alter the convergence behavior of stochastic gradient descent (SGD). In this work, we study the convergence of SGD under a gradient shrinkage model, where each stochastic gradient is scaled by a factor $q_k \in (0,1]$ and perturbed by zero-mean quantization noise. We show that this shrinkage is equivalent to replacing the nominal stepsize $\mu_k$ with an effective stepsize $\mu_k q_k$, which slows convergence when $q_{\min} < 1$. Under standard smoothness and bounded-variance assumptions, we prove that low-precision SGD still converges, but at a reduced rate determined by $q_{\min}$, and with an increased asymptotic error floor due to quantization noise. We theoretically analyze how reduced numerical precision slows down training by modeling it as gradient shrinkage in the standard SGD convergence framework.
Abstract (translated)
低精度训练已成为减少大规模深度学习计算和内存成本的关键方法。然而，梯度量化会引入幅度衰减以及加性噪声，这些都可能改变随机梯度下降（SGD）的收敛行为。在这项工作中，我们研究了在一种梯度缩放模型下的SGD收敛性，在这种模型中，每个随机梯度被一个因子$q_k \in (0,1]$ 缩小，并且受到零均值量化噪声的影响。 我们的研究表明，该缩放等同于将名义学习率$\mu_k$替换为有效学习率$\mu_k q_k$。当最小的缩放因子$q_{\min} < 1$时，这会导致收敛速度变慢。在标准平滑性和有界方差假设下，我们证明了低精度SGD仍然能够收敛，但以一个由$q_{\min}$决定的速度减缓，并且由于量化噪声导致最终误差增加。 从理论上讲，我们通过将数值精度降低建模为标准SGD收敛框架中的梯度缩放来分析其如何减慢训练速度。
URL
https://arxiv.org/abs/2508.07142
