Abstract
The latest deep learning-based approaches have shown promising results for the challenging task of inpainting missing regions of an image. However, the existing methods often generate contents with blurry textures and distorted structures due to the discontinuity of the local pixels. From a semantic-level perspective, the local pixel discontinuity is mainly because these methods ignore the semantic relevance and feature continuity of hole regions. To handle this problem, we investigate the human behavior in repairing pictures and propose a fined deep generative model-based approach with a novel coherent semantic attention (CSA) layer, which can not only preserve contextual structure but also make more effective predictions of missing parts by modeling the semantic relevance between the holes features. The task is divided into rough, refinement as two steps and model each step with a neural network under the U-Net architecture, where the CSA layer is embedded into the encoder of refinement step. To stabilize the network training process and promote the CSA layer to learn more effective parameters, we propose a consistency loss to enforce the both the CSA layer and the corresponding layer of the CSA in decoder to be close to the VGG feature layer of a ground truth image simultaneously. The experiments on CelebA, Places2, and Paris StreetView datasets have validated the effectiveness of our proposed methods in image inpainting tasks and can obtain images with a higher quality as compared with the existing state-of-the-art approaches.
Abstract (translated)
最新的基于深度学习的方法对于修复图像缺失区域的挑战性任务显示出了良好的效果。然而，由于局部像素的不连续性，现有的方法往往会生成纹理模糊、结构扭曲的内容。从语义层面上看，局部像素的不连续性主要是由于这些方法忽略了空穴区域的语义相关性和特征连续性。为了解决这一问题，我们对修复图像中的人类行为进行了研究，提出了一种基于深度生成模型的精细方法，该方法采用了一种新的相干语义注意（CSA）层，该层不仅可以保留上下文结构，而且可以通过建模孔之间的语义相关性来更有效地预测缺失的部分。S功能。该任务分为粗、精、精两步，在U-NET架构下，用神经网络对每一步进行建模，在U-NET架构下，CSA层嵌入精、精编码步骤。为了稳定网络训练过程，促进CSA层学习更有效的参数，我们提出了一种一致性损失，使译码器中的CSA层和相应的CSA层同时接近地面真值图像的VGG特征层。通过对Celeba、Places2和Paris StreetView数据集的实验，验证了我们提出的方法在图像修复任务中的有效性，与现有的最先进的方法相比，可以获得更高质量的图像。
URL
https://arxiv.org/abs/1905.12384
