Abstract
Diffusion models have emerged as a key pillar of foundation models in visual domains. One of their critical applications is to universally solve different downstream inverse tasks via a single diffusion prior without re-training for each task. Most inverse tasks can be formulated as inferring a posterior distribution over data (e.g., a full image) given a measurement (e.g., a masked image). This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution. We show that our approach naturally leads to regularization by denoising diffusion process (RED-Diff) where denoisers at different timesteps concurrently impose different structural constraints over the image. To gauge the contribution of denoisers from different timesteps, we propose a weighting mechanism based on signal-to-noise-ratio (SNR). Our approach provides a new variational perspective for solving inverse problems with diffusion models, allowing us to formulate sampling as stochastic optimization, where one can simply apply off-the-shelf solvers with lightweight iterates. Our experiments for image restoration tasks such as inpainting and superresolution demonstrate the strengths of our method compared with state-of-the-art sampling-based diffusion models.
Abstract (translated)
扩散模型已成为视觉领域的基础模型之一，其关键应用是普遍通过一个扩散先验解决不同下游反演任务，而无需对每个任务进行重新训练。大多数反演任务可以表示为推断数据分布(例如，全图像)给定测量(例如，掩膜图像)的概率分布。然而，在扩散模型中，这仍然是一个挑战，因为扩散过程非线性和迭代的性质导致概率分布难以估计。为了应对这一挑战，我们提出了一种Variational 方法，其设计旨在近似真正的后概率分布。我们证明，我们的方法自然地通过去噪扩散过程 regularize (RED-Diff)，其中在不同时间步长上的信号噪声比(SNR)作为权重机制对图像进行加权。为了衡量不同时间步长上的去噪器的贡献，我们提出了一种基于随机优化的加权机制。我们的方法提供了解决扩散模型的反演问题的新Variational 视角，使我们能够将其采样作为随机优化， where one can simply apply off-the-shelf solving algorithms with lightweight iterates.我们对图像修复任务，例如填空和超分辨率，进行了实验，证明了我们方法与最先进的基于采样扩散模型相比的优势。
URL
https://arxiv.org/abs/2305.04391
