Abstract
Diffusion models have shown promising results in free-form inpainting. Recent studies based on refined diffusion samplers or novel architectural designs led to realistic results and high data consistency. However, random initialization seed (noise) adopted in vanilla diffusion process may introduce mismatched semantic information in masked regions, leading to biased inpainting results, e.g., low consistency and low coherence with the other unmasked area. To address this issue, we propose the Initial Seed refined Diffusion Model (IS-Diff), a completely training-free approach incorporating distributional harmonious seeds to produce harmonious results. Specifically, IS-Diff employs initial seeds sampled from unmasked areas to imitate the masked data distribution, thereby setting a promising direction for the diffusion procedure. Moreover, a dynamic selective refinement mechanism is proposed to detect severe unharmonious inpaintings in intermediate latent and adjust the strength of our initialization prior dynamically. We validate our method on both standard and large-mask inpainting tasks using the CelebA-HQ, ImageNet, and Places2 datasets, demonstrating its effectiveness across all metrics compared to state-of-the-art inpainting methods.
Abstract (translated)
扩散模型在自由形式的图像修复（inpainting）中展现出了很有前景的结果。近期基于改进的扩散采样器或新颖架构设计的研究，产生了逼真的结果和高度的数据一致性。然而，在标准的扩散过程中采用随机初始化种子（噪声），可能会导致掩码区域中的语义信息不匹配，从而产生有偏差的修复结果，例如与未掩码区域的一致性和连贯性较差。 为了解决这个问题，我们提出了初始种子优化扩散模型（Initial Seed refined Diffusion Model, IS-Diff）。这是一种完全无需训练的方法，通过采用分布和谐化的种子来生成和谐的结果。具体来说，IS-Diff 使用从未掩码区域采样的初始种子，以模仿被遮挡数据的分布，并为扩散过程设定一个有前景的方向。此外，我们提出了一种动态选择性细化机制，用于检测中间潜在层中的严重不和谐修复结果，并根据实际情况动态调整初始化先验的强度。 我们在CelebA-HQ、ImageNet和Places2数据集上对标准和大面积掩码图像修复任务进行了验证，结果显示了IS-Diff在所有评估指标上的优越性能，优于现有的最先进的图像修复方法。
URL
https://arxiv.org/abs/2509.11638
