Abstract
Quantization lowers memory usage, computational requirements, and latency by utilizing fewer bits to represent model weights and activations. In this work, we investigate the generalization properties of quantized neural networks, a characteristic that has received little attention despite its implications on model performance. In particular, first, we develop a theoretical model for quantization in neural networks and demonstrate how quantization functions as a form of regularization. Second, motivated by recent work connecting the sharpness of the loss landscape and generalization, we derive an approximate bound for the generalization of quantized models conditioned on the amount of quantization noise. We then validate our hypothesis by experimenting with over 2000 models trained on CIFAR-10, CIFAR-100, and ImageNet datasets on convolutional and transformer-based models.
Abstract (translated)
量化降低内存使用、计算要求和延迟，通过使用更少的比特来表示模型权重和激活。在这项工作中，我们研究了量化神经网络的泛化特性，尽管这对模型性能有着深刻的意义，但这个特性并未受到太多的关注。特别是，我们为神经网络的量化开发了一个理论模型，并证明了量化作为一种正则化形式的作用。第二，为了研究最近工作连接损失函数的尖度与泛化之间的关系，我们推导了量化模型的泛化近界，基于量化噪声的数量。然后，我们通过在CIFAR-10、CIFAR-100和ImageNet数据集上使用卷积和Transformer基模型训练超过2000个模型来验证我们的假设。
URL
https://arxiv.org/abs/2404.11769
