Abstract
Recent diffusion-based models achieve photorealism in image inpainting but require many sampling steps, limiting practical use. Few-step text-to-image models offer faster generation, but naively applying them to inpainting yields poor harmonization and artifacts between the background and inpainted region. We trace this cause to random Gaussian noise initialization, which under low function evaluations causes semantic misalignment and reduced fidelity. To overcome this, we propose InverFill, a one-step inversion method tailored for inpainting that injects semantic information from the input masked image into the initial noise, enabling high-fidelity few-step inpainting. Instead of training inpainting models, InverFill leverages few-step text-to-image models in a blended sampling pipeline with semantically aligned noise as input, significantly improving vanilla blended sampling and even matching specialized inpainting models at low NFEs. Moreover, InverFill does not require real-image supervision and only adds minimal inference overhead. Extensive experiments show that InverFill consistently boosts baseline few-step models, improving image quality and text coherence without costly retraining or heavy iterative optimization.
Abstract (translated)
近期基于扩散的模型在图像修复中已能实现照片级真实感，但需要大量采样步骤，限制了实际应用。少步数文生图模型虽能加速生成，但直接将其用于修复会导致背景与修复区域间融合不佳并产生伪影。我们追溯此问题根源在于随机高斯噪声初始化——在低函数评估次数下，这会导致语义错位并降低保真度。为克服此问题，我们提出InverFill，一种专为修复设计的一步反转方法，它能将输入掩码图像的语义信息注入初始噪声，从而实现高保真度的少步数修复。InverFill无需训练修复模型，而是通过语义对齐噪声作为输入，在混合采样流程中利用少步数文生图模型，显著改进了原始混合采样，甚至在低函数评估次数下能媲美专业修复模型。此外，InverFill无需真实图像监督，且仅增加极小的推理开销。大量实验表明，InverFill能持续提升基线少步数模型，改善图像质量与文本一致性，且无需代价高昂的重新训练或繁重的迭代优化。
URL
https://arxiv.org/abs/2603.23463
