Abstract
The rising performance of deep neural networks is often empirically attributed to an increase in the available computational power, which allows complex models to be trained upon large amounts of annotated data. However, increased model complexity leads to costly deployment of modern neural networks, while gathering such amounts of data requires huge costs to avoid label noise. In this work, we study the ability of compression methods to tackle both of these problems at once. We hypothesize that quantization-aware training, by restricting the expressivity of neural networks, behaves as a regularization. Thus, it may help fighting overfitting on noisy data while also allowing for the compression of the model at inference. We first validate this claim on a controlled test with manually introduced label noise. Furthermore, we also test the proposed method on Facial Action Unit detection, where labels are typically noisy due to the subtlety of the task. In all cases, our results suggests that quantization significantly improve the results compared with existing baselines, regularization as well as other compression methods.
Abstract (translated)
深度学习模型的性能不断提升往往Empirically归咎于可用的计算资源增加,这使得复杂的模型能够基于大量标注数据进行训练。然而,模型复杂性的增加会导致现代神经网络的昂贵部署,而收集这样数量的数据需要巨大的成本以避免标签噪声。在本文中,我们将研究压缩方法如何解决这两个问题同时存在。我们假设有意识训练可以通过限制神经网络的表达力来被视为正则化。因此,它可能有助于在噪声数据上避免过拟合,同时也允许模型在推理时进行压缩。我们首先在手动引入标签噪声的控制测试中验证这一假设。此外,我们还测试了提出的方法,用于面部行动单元检测,该任务通常由于任务的微妙性而存在标签噪声。在所有情况下,我们的结果表明,量化 significantly improve 结果 compared with existing baselines, regularization as well as other compression methods.
URL
https://arxiv.org/abs/2303.11803
