Abstract
Text-guided image inpainting endeavors to generate new content within specified regions of images using textual prompts from users. The primary challenge is to accurately align the inpainted areas with the user-provided prompts while maintaining a high degree of visual fidelity. While existing inpainting methods have produced visually convincing results by leveraging the pre-trained text-to-image diffusion models, they still struggle to uphold both prompt alignment and visual rationality simultaneously. In this work, we introduce FreeInpaint, a plug-and-play tuning-free approach that directly optimizes the diffusion latents on the fly during inference to improve the faithfulness of the generated images. Technically, we introduce a prior-guided noise optimization method that steers model attention towards valid inpainting regions by optimizing the initial noise. Furthermore, we meticulously design a composite guidance objective tailored specifically for the inpainting task. This objective efficiently directs the denoising process, enhancing prompt alignment and visual rationality by optimizing intermediate latents at each step. Through extensive experiments involving various inpainting diffusion models and evaluation metrics, we demonstrate the effectiveness and robustness of our proposed FreeInpaint.
Abstract (translated)
文本引导的图像修复技术旨在通过用户的文本提示，在指定区域内生成新的内容。主要挑战在于确保修复区域与用户提供的提示准确对齐，同时保持高度的视觉真实性。尽管现有的图像修复方法已经利用预训练的文本到图像扩散模型取得了令人信服的视觉效果，但它们在同时维持提示对齐和视觉合理性方面仍面临困难。 在此项工作中，我们引入了FreeInpaint，这是一种即插即用且无需调整的方法，它通过在推理过程中直接优化扩散潜变量来提高生成图像的真实性。技术上，我们提出了一种引导先验的噪声优化方法，该方法通过对初始噪声进行优化，使模型注意力集中在有效的修复区域上。此外，我们精心设计了一个复合引导目标，专门针对图像修复任务。这一目标可以有效指导去噪过程，在每一步通过优化中间潜变量来增强提示对齐和视觉合理性。 通过涉及各种图像修复扩散模型及评估指标的广泛实验，我们展示了所提出的FreeInpaint的有效性和鲁棒性。
URL
https://arxiv.org/abs/2512.21104
