Abstract
A popular track of network compression approach is Quantization aware Training (QAT), which accelerates the forward pass during the neural network training and inference. However, not much prior efforts have been made to quantize and accelerate the backward pass during training, even though that contributes around half of the training time. This can be partly attributed to the fact that errors of low-precision gradients during backward cannot be amortized by the training objective as in the QAT setting. In this work, we propose to solve this problem by incorporating the gradients into the computation graph of the next training iteration via a hypernetwork. Various experiments on CIFAR-10 dataset with different CNN network architectures demonstrate that our hypernetwork-based approach can effectively reduce the negative effect of gradient quantization noise and successfully quantizes the gradients to INT4 with only 0.64 accuracy drop for VGG-16 on CIFAR-10.
Abstract (translated)
网络压缩方法的常见路径是量化意识到的训练(QAT),这加速了神经网络训练和推理的前端流程。然而,尽管在训练过程中量化和加速后端流程的贡献约占一半,但几乎没有先前的努力来量化和加速训练过程中后端流程。这部分地归咎于事实,即后端低精度梯度的错误在backward过程中无法像QAT设置那样被训练目标所抵消。在本研究中,我们建议通过引入梯度到下一个训练迭代的计算图中并通过超网络来实现。各种CIFAR-10数据集的不同卷积神经网络架构的实验表明,我们的超网络方法可以有效地减少梯度量化噪声的负面影响,并将梯度成功地量化到INT4,在CIFAR-10中只有VGG-16的精度下降达到了0.64。
URL
https://arxiv.org/abs/2303.02347
