Abstract
Recently, deep learning has become a de facto standard in machine learning with convolutional neural networks (CNNs) demonstrating spectacular success on a wide variety of tasks. However, CNNs are typically very demanding computationally at inference time. One of the ways to alleviate this burden on certain hardware platforms is quantization relying on the use of low-precision arithmetic representation for the weights and the activations. Another popular method is the pruning of the number of filters in each layer. While mainstream deep learning methods train the neural networks weights while keeping the network architecture fixed, the emerging neural architecture search (NAS) techniques make the latter also amenable to training. In this paper, we formulate optimal arithmetic bit length allocation and neural network pruning as a NAS problem, searching for the configurations satisfying a computational complexity budget while maximizing the accuracy. We use a differentiable search method based on the continuous relaxation of the search space proposed by Liu et al. (<a href="https://export.arxiv.org/abs/1806.09055">arXiv:1806.09055</a>). We show, by grid search, that heterogeneous quantized networks suffer from a high variance which renders the benefit of the search questionable. For pruning, improvement over homogeneous cases is possible, but it is still challenging to find those configurations with the proposed method. The code is publicly available at https://github.com/yochaiz/Slimmable and https://github.com/yochaiz/darts-UNIQ .
Abstract (translated)
最近，深度学习已经成为机器学习的一个事实标准，卷积神经网络（CNN）在各种各样的任务上显示出惊人的成功。然而，CNN在推理时通常要求非常严格的计算。在某些硬件平台上减轻这种负担的方法之一是量化，它依赖于对权重和激活使用低精度算术表示。另一种流行的方法是修剪每层中的过滤器数量。虽然主流的深度学习方法训练神经网络权重，同时保持网络结构不变，但新兴的神经结构搜索（NAS）技术也使后者易于训练。本文将优化算法比特长度分配和神经网络剪枝作为一个NAS问题，在最大化精度的同时寻找满足计算复杂度预算的配置。我们使用刘等人提出的基于搜索空间连续松弛的可微搜索方法。（<a href=“https://export.arxiv.org/abs/1806.09055”>arxiv:1806.09055<a>）。通过网格搜索，我们发现异构量化网络存在很大的方差，这使得搜索的好处受到质疑。对于修剪来说，对齐次情况的改进是可能的，但是用所提出的方法找到那些配置仍然是困难的。该代码可在https://github.com/yochaiz/slimmable和https://github.com/yochaiz/darts-uniq上公开获取。
URL
https://arxiv.org/abs/1904.09872
