Abstract
Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is true even for the simplest cases that involve concave log-likelihoods and use a Gaussian approximation. We show that the challenge can be circumvented for such cases using a square-root parameterization for the Gaussian covariance. This approach establishes novel convergence guarantees for natural-gradient variational-Gaussian inference and its continuous-time gradient flow. Our experiments demonstrate the effectiveness of natural gradient methods and highlight their advantages over algorithms that use Euclidean or Wasserstein geometries.
Abstract (translated)
基于自然梯度下降的变分推理在实践中常常表现出快速收敛的特点，但其理论上的收敛保证却难以确立。即使是最简单的情况——涉及凹对数似然函数并使用高斯近似的情形也是如此。我们展示了在这种情况下，可以通过为高斯协方差采用平方根参数化来克服这一挑战。这种方法建立了自然梯度变分-高斯推理及其连续时间梯度流的新收敛保证。我们的实验表明了自然梯度方法的有效性，并强调了它们相对于使用欧几里得或 Wasserstein 几何算法的优势。
URL
https://arxiv.org/abs/2507.07853
