Abstract
The ability of overparameterized deep networks to generalize well has been linked to the fact that stochastic gradient descent (SGD) finds solutions that lie in flat, wide minima in the training loss -- minima where the output of the network is resilient to small random noise added to its parameters. So far this observation has been used to provide generalization guarantees only for neural networks whose parameters are either \textit{stochastic} or \textit{compressed}. In this work, we present a general PAC-Bayesian framework that leverages this observation to provide a bound on the original network learned -- a network that is deterministic and uncompressed. What enables us to do this is a key novelty in our approach: our framework allows us to show that if on training data, the interactions between the weight matrices satisfy certain conditions that imply a wide training loss minimum, these conditions themselves {\em generalize} to the interactions between the matrices on test data, thereby implying a wide test loss minimum. We then apply our general framework in a setup where we assume that the pre-activation values of the network are not too small (although we assume this only on the training data). In this setup, we provide a generalization guarantee for the original (deterministic, uncompressed) network, that does not scale with product of the spectral norms of the weight matrices -- a guarantee that would not have been possible with prior approaches.
Abstract (translated)
多帧深网络推广良好的能力与以下事实有关：随机梯度下降（SGD）在训练损失中找到平坦、宽的最小值——最小值，其中网络输出对添加到其参数中的小随机噪声具有弹性。到目前为止，这项研究仅用于为参数为 extit随机或 extit压缩的神经网络提供泛化保证。在这项工作中，我们提出了一个通用的PAC贝叶斯框架，它利用这一观察结果提供了原始网络学习的一个边界——一个确定性和未压缩的网络。使我们能够做到这一点的是我们方法的一个关键新颖之处：我们的框架允许我们证明，如果在训练数据上，权重矩阵之间的交互满足某些条件，这意味着训练损失最小，这些条件本身em将归纳为测试数据矩阵之间的交互，从而暗示测试损失最小。然后，我们在一个设置中应用我们的通用框架，在这个设置中，我们假设网络的预激活值不太小（尽管我们只在培训数据上假设）。在这个设置中，我们为原始的（确定性的，未压缩的）网络提供了一个泛化保证，它不随权重矩阵的光谱规范的乘积而缩放——这是一个在先前的方法中不可能实现的保证。
URL
https://arxiv.org/abs/1905.13344
