Abstract
Text-guided image inpainting aims at reconstructing the masked regions as per text prompts, where the longstanding challenges lie in the preservation for unmasked regions, while achieving the semantics consistency between unmasked and inpainted masked regions. Previous arts failed to address both of them, always with either of them to be remedied. Such facts, as we observed, stem from the entanglement of the hybrid (e.g., mid-and-low) frequency bands that encode varied image properties, which exhibit different robustness to text prompts during the denoising process. In this paper, we propose a null-text-null frequency-aware diffusion models, dubbed \textbf{NTN-Diff}, for text-guided image inpainting, by decomposing the semantics consistency across masked and unmasked regions into the consistencies as per each frequency band, while preserving the unmasked regions, to circumvent two challenges in a row. Based on the diffusion process, we further divide the denoising process into early (high-level noise) and late (low-level noise) stages, where the mid-and-low frequency bands are disentangled during the denoising process. As observed, the stable mid-frequency band is progressively denoised to be semantically aligned during text-guided denoising process, which, meanwhile, serves as the guidance to the null-text denoising process to denoise low-frequency band for the masked regions, followed by a subsequent text-guided denoising process at late stage, to achieve the semantics consistency for mid-and-low frequency bands across masked and unmasked regions, while preserve the unmasked regions. Extensive experiments validate the superiority of NTN-Diff over the state-of-the-art diffusion models to text-guided diffusion models. Our code can be accessed from this https URL.
Abstract (translated)
文本引导的图像修复（Text-guided image inpainting）旨在根据文本提示重建被遮罩区域，其中长期存在的挑战在于保持未遮罩区域的同时实现遮罩与非遮罩区域之间的语义一致性。先前的方法未能同时解决这两个问题，总是在两者之间做出取舍。我们注意到这种现象的原因是混合（例如中频和低频）频带的纠缠，这些频带编码了不同的图像属性，在去噪过程中对文本提示展现出不同级别的鲁棒性。 为了应对这一挑战，我们在本文中提出了一种基于零文本-零频率感知扩散模型的方法，称为**NTN-Diff**。该方法通过将遮罩区域与非遮罩区域之间的语义一致性分解为各个频带的一致性，并保持未被遮盖的区域来依次解决这两个问题。 在扩散过程中，我们将去噪过程进一步分为早期（高层噪声）和晚期（低层噪声）两个阶段，在这一过程中分离了中频和低频。我们观察到，在文本引导的去噪处理过程中，稳定的中频带逐步减少噪声并变得语义对齐，并且它充当指导零文本去噪的过程以去除遮罩区域的低频噪声，随后在晚期阶段进行后续的文本引导去噪过程，实现被掩码与非掩码区域之间跨中、低频带的语义一致性同时保持未遮蔽区域。 广泛的实验验证了NTN-Diff相比现有的最先进的扩散模型在基于文本的图像修复任务上的优越性。我们的代码可以从这个链接访问：[请在此处插入实际的URL]。
URL
https://arxiv.org/abs/2510.08273
