Abstract
Model parameter regularization is a widely used technique to improve generalization, but also can be used to shape the weight distributions for various purposes. In this work, we shed light on how weight regularization can assist model quantization and compression techniques, and then propose range regularization (R^2) to further boost the quality of model optimization by focusing on the outlier prevention. By effectively regulating the minimum and maximum weight values from a distribution, we mold the overall distribution into a tight shape so that model compression and quantization techniques can better utilize their limited numeric representation powers. We introduce L-inf regularization, its extension margin regularization and a new soft-min-max regularization to be used as a regularization loss during full-precision model training. Coupled with state-of-the-art quantization and compression techniques, models trained with R^2 perform better on an average, specifically at lower bit weights with 16x compression ratio. We also demonstrate that R^2 helps parameter constrained models like MobileNetV1 achieve significant improvement of around 8% for 2 bit quantization and 7% for 1 bit compression.
Abstract (translated)
模型参数Regularization是一种广泛应用的技术,以改善泛化能力,但也可用于形状权重分布的各种目的。在本文中,我们阐明了如何帮助模型量化和压缩技术,然后提出了范围Regularization(R^2),以通过重点预防异常点来进一步提高模型优化的质量。通过有效地管理分布中的最小和最大值权重值,我们塑造了整个分布的紧凑形状,使模型量化和编码技术更好地利用其有限的数字表示能力。我们引入了L-inf Regularization,其扩展margin regularization和新的弹性最小最大值 Regularization,并在全精度模型训练期间用作 Regularization 损失。与最先进的量化和压缩技术相结合,训练使用R^2的模型平均表现更好,特别是低比特权重并具有16x压缩比时。我们还证明R^2帮助约束参数的模型如MobileNetV1实现约8%的2比特量化和1比特压缩的显著改进。
URL
https://arxiv.org/abs/2303.08253
