Abstract
This paper presents a new adversarial training framework for image inpainting with segmentation confusion adversarial training (SCAT) and contrastive learning. SCAT plays an adversarial game between an inpainting generator and a segmentation network, which provides pixel-level local training signals and can adapt to images with free-form holes. By combining SCAT with standard global adversarial training, the new adversarial training framework exhibits the following three advantages simultaneously: (1) the global consistency of the repaired image, (2) the local fine texture details of the repaired image, and (3) the flexibility of handling images with free-form holes. Moreover, we propose the textural and semantic contrastive learning losses to stabilize and improve our inpainting model's training by exploiting the feature representation space of the discriminator, in which the inpainting images are pulled closer to the ground truth images but pushed farther from the corrupted images. The proposed contrastive losses better guide the repaired images to move from the corrupted image data points to the real image data points in the feature representation space, resulting in more realistic completed images. We conduct extensive experiments on two benchmark datasets, demonstrating our model's effectiveness and superiority both qualitatively and quantitatively.
Abstract (translated)
本论文提出了一种用于图像修复与分割混淆GAN训练(SCAT)和对比学习的新对抗训练框架。SCAT是一个修复生成器和分割网络之间的对抗游戏，它提供像素级别的局部训练信号，并能够适应具有自由漏洞的图像。通过将SCAT与标准全局GAN训练相结合，新框架同时表现出以下三个优点：(1)修复图像的全局一致性，(2)修复图像的局部细节特征，(3)能够灵活处理具有自由漏洞的图像。此外，我们提出了纹理和语义对比学习损失，以稳定和提高修复模型的训练，利用分选器的特征表示空间，在修复图像靠近 ground truth 图像但远离损坏图像的情况下，将修复图像从损坏图像数据点移动到特征表示空间的真实图像数据点，从而生成更真实的完成图像。我们在两个基准数据集上进行了广泛的实验，证明了我们的模型的有效性和优越性，既定性上也定量上。
URL
https://arxiv.org/abs/2303.13133
