Abstract
Advancements in generative models have enabled image inpainting models to generate content within specific regions of an image based on provided prompts and masks. However, existing inpainting methods often suffer from problems such as semantic misalignment, structural distortion, and style inconsistency. In this work, we present MTADiffusion, a Mask-Text Alignment diffusion model designed for object inpainting. To enhance the semantic capabilities of the inpainting model, we introduce MTAPipeline, an automatic solution for annotating masks with detailed descriptions. Based on the MTAPipeline, we construct a new MTADataset comprising 5 million images and 25 million mask-text pairs. Furthermore, we propose a multi-task training strategy that integrates both inpainting and edge prediction tasks to improve structural stability. To promote style consistency, we present a novel inpainting style-consistency loss using a pre-trained VGG network and the Gram matrix. Comprehensive evaluations on BrushBench and EditBench demonstrate that MTADiffusion achieves state-of-the-art performance compared to other methods.
Abstract (translated)
生成模型的进步使图像修复（inpainting）模型能够在特定区域根据提供的提示和蒙版生成内容。然而，现有的修复方法常常面临语义错配、结构扭曲和风格不一致等问题。为此，我们提出了一种名为MTADiffusion的掩码-文本对齐扩散模型，专为对象修复设计。 为了增强修复模型的语义能力，我们引入了自动解决方案MTAPipeline，用于用详细的描述来标注蒙版。基于MTAPipeline，我们构建了一个新的MTADataset，包括500万张图像和2500万个掩码-文本对。 此外，我们提出了一种多任务训练策略，该策略结合修复和边缘预测任务以提高结构稳定性。为了促进风格一致性，我们提出了一种新颖的修复风格一致性损失方法，使用预训练的VGG网络和Gram矩阵来实现。 在BrushBench和EditBench上的全面评估表明，MTADiffusion相对于其他方法实现了最先进的性能。
URL
https://arxiv.org/abs/2506.23482
