Abstract
We establish an equivalence between information bottleneck (IB) learning and an unconventional quantization problem, `IB quantization'. Under this equivalence, standard neural network models correspond to scalar IB quantizers. We prove a coding theorem for IB quantization, which implies that scalar IB quantizers are in general inferior to vector IB quantizers. This inspires us to develop a learning framework for neural networks, AgrLearn, that corresponds to vector IB quantizers. We experimentally verify that AgrLearn applied to some deep network models of current art improves upon them, while requiring less training data. With a heuristic smoothing, AgrLearn further improves its performance, resulting in new state of the art in image classification on Cifar10.
Abstract (translated)
我们在信息瓶颈（IB）学习和非常规量化问题“IB量化”之间建立等价。在这种等价性下，标准神经网络模型对应于标量IB量化器。我们证明了IB量化的编码定理，这意味着标量IB量化器通常不如矢量IB量化器。这激发了我们开发神经网络的学习框架，AgrLearn，它对应于矢量IB量化器。我们通过实验验证了AgrLearn应用于当前艺术的一些深度网络模型的改进，同时需要较少的训练数据。通过启发式平滑，AgrLearn进一步提高了性能，从而在Cifar10上实现了图像分类的最新技术水平。
URL
https://arxiv.org/abs/1807.10251
