Abstract
Modern deep learning models generalize remarkably well in-distribution, despite being overparametrized and trained with little to no explicit regularization. Instead, current theory credits implicit regularization imposed by the choice of architecture, hyperparameters and optimization procedure. However, deploying deep learning models out-of-distribution, in sequential decision-making tasks, or in safety-critical domains, necessitates reliable uncertainty quantification, not just a point estimate. The machinery of modern approximate inference -- Bayesian deep learning -- should answer the need for uncertainty quantification, but its effectiveness has been challenged by our inability to define useful explicit inductive biases through priors, as well as the associated computational burden. Instead, in this work we demonstrate, both theoretically and empirically, how to regularize a variational deep network implicitly via the optimization procedure, just as for standard deep learning. We fully characterize the inductive bias of (stochastic) gradient descent in the case of an overparametrized linear model as generalized variational inference and demonstrate the importance of the choice of parametrization. Finally, we show empirically that our approach achieves strong in- and out-of-distribution performance without tuning of additional hyperparameters and with minimal time and memory overhead over standard deep learning.
Abstract (translated)
现代深度学习模型在分布内情况下表现出色，尽管它们参数过多且训练过程中几乎没有显式的正则化。当前理论认为，架构的选择、超参数和优化过程所隐含的规则化是关键因素。然而，在部署这些模型以处理分布外数据、顺序决策任务或安全攸关领域时，可靠的不确定性量化比单一的点估计更为重要。现代近似推理工具——贝叶斯深度学习——应能满足这种对不确定性的需求，但其有效性因我们难以通过先验定义有用的显式归纳偏置以及由此产生的计算负担而受到质疑。 在这项工作中，我们理论和实证地展示了如何通过优化过程隐式正则化变分深层网络，就像标准的深度学习一样。我们在超参数化的线性模型的情况下全面描述了(随机)梯度下降的归纳偏差，并将其归类为广义变分推理。此外，还证明了选择参数化方式的重要性。 最后，我们通过实验展示了我们的方法能够在不调整额外超参数且仅增加少量时间和内存开销的情况下实现强大的分布内和分布外性能。
URL
https://arxiv.org/abs/2505.20235
