Abstract
We highlight a fundamental connection between information geometry and variational Bayes (VB) and discuss its consequences for machine learning. Under certain conditions, a VB solution always requires estimation or computation of natural gradients. We show several consequences of this fact by using the natural-gradient descent algorithm of Khan and Rue (2023) called the Bayesian Learning Rule (BLR). These include (i) a simplification of Bayes' rule as addition of natural gradients, (ii) a generalization of quadratic surrogates used in gradient-based methods, and (iii) a large-scale implementation of VB algorithms for large language models. Neither the connection nor its consequences are new but we further emphasize the common origins of the two fields of information geometry and Bayes with a hope to facilitate more work at the intersection of the two fields.
Abstract (translated)
我们强调了信息几何与变分贝叶斯（VB）之间的基本联系，并讨论了这对机器学习的含义。在某些条件下，VB解总是需要估计或计算自然梯度。通过使用Khan和Rue (2023)提出的称为贝叶斯学习规则(BLR)的自然梯度下降算法，我们展示了这一事实的几个后果。这包括(i) 贝叶斯法则作为自然梯度相加的简化形式；(ii) 对基于梯度方法中使用的二次代理的推广；以及(iii) 大规模实现适用于大型语言模型的VB算法。虽然这种联系及其影响并不新鲜，但我们进一步强调了信息几何和贝叶斯这两个领域共同起源的重要性，并希望这能够促进两个领域交叉点上更多工作的开展。
URL
https://arxiv.org/abs/2509.15641
