Abstract
Amortized variational inference (AVI) replaces instance-specific local inference with a global inference network. While AVI has enabled efficient training of deep generative models such as variational autoencoders (VAE), recent empirical work suggests that inference networks can produce suboptimal variational parameters. We propose a hybrid approach, to use AVI to initialize the variational parameters and run stochastic variational inference (SVI) to refine them. Crucially, the local SVI procedure is itself differentiable, so the inference network and generative model can be trained end-to-end with gradient-based optimization. This semi-amortized approach enables the use of rich generative models without experiencing the posterior-collapse phenomenon common in training VAEs for problems like text generation. Experiments show this approach outperforms strong autoregressive and variational baselines on standard text and image datasets.
Abstract (translated)
分摊变分推理（AVI）用全局推理网络替换特定于实例的局部推理。虽然AVI已经能够对变分自动编码器（VAE）等深度生成模型进行有效的训练，但最近的实证研究表明，推理网络可以产生次优的变分参数。我们提出了一种混合方法，即使用AVI初始化变分参数并运行随机变分推理（SVI）来优化它们。至关重要的是，局部SVI程序本身是可微分的，因此推理网络和生成模型可以通过基于梯度的优化进行端到端训练。这种半摊销方法可以使用丰富的生成模型，而不会遇到训练VAE中常见的后塌陷现象，如文本生成等问题。实验表明，这种方法在标准文本和图像数据集上优于强自回归和变分基线。
URL
https://arxiv.org/abs/1802.02550
