Abstract
The use of low-bit quantization has emerged as an indispensable technique for enabling the efficient training of large-scale models. Despite its widespread empirical success, a rigorous theoretical understanding of its impact on learning performance remains notably absent, even in the simplest linear regression setting. We present the first systematic theoretical study of this fundamental question, analyzing finite-step stochastic gradient descent (SGD) for high-dimensional linear regression under a comprehensive range of quantization targets: data, labels, parameters, activations, and gradients. Our novel analytical framework establishes precise algorithm-dependent and data-dependent excess risk bounds that characterize how different quantization affects learning: parameter, activation, and gradient quantization amplify noise during training; data quantization distorts the data spectrum; and data and label quantization introduce additional approximation and quantized error. Crucially, we prove that for multiplicative quantization (with input-dependent quantization step), this spectral distortion can be eliminated, and for additive quantization (with constant quantization step), a beneficial scaling effect with batch size emerges. Furthermore, for common polynomial-decay data spectra, we quantitatively compare the risks of multiplicative and additive quantization, drawing a parallel to the comparison between FP and integer quantization methods. Our theory provides a powerful lens to characterize how quantization shapes the learning dynamics of optimization algorithms, paving the way to further explore learning theory under practical hardware constraints.
Abstract (translated)
低比特量化作为一种技术，已成为实现大规模模型高效训练不可或缺的方法。尽管它在经验上取得了广泛的成功，但在最简单的线性回归设置中，对其对学习性能影响的严格理论理解仍然明显缺乏。我们首次系统地研究了这一基本问题，分析了高维线性回归下的有限步随机梯度下降（SGD），涵盖了数据、标签、参数、激活和梯度等全面范围内的量化目标。我们的新型分析框架建立了一套精确的算法依赖性和数据依赖性的额外风险界限，这些界限描述了不同类型的量化如何影响学习过程：参数量化、激活量化和梯度量化在训练过程中放大噪声；数据量化扭曲数据谱；而数据和标签量化则引入额外的近似误差和量化误差。尤为重要的是，我们证明对于乘法量化（具有输入依赖性的量化步长），这种光谱失真可以被消除，并且对于加性量化（具有恒定的量化步长），随着批量大小的增加会形成有益的缩放效应。此外，在常见的多项式衰减数据谱中，我们从定量上比较了乘法和加性量化的风险，将其与浮点数和整数量化方法之间的比较进行了类比。我们的理论为如何通过量化塑造优化算法的学习动态提供了有力的研究视角，并为进一步在实际硬件约束下的学习理论研究铺平了道路。
URL
https://arxiv.org/abs/2510.18259
