Abstract
Facial Image inpainting aim is to restore the missing or corrupted regions in face images while preserving identity, structural consistency and photorealistic image quality, a task specifically created for photo restoration. Though there are recent lot of advances in deep generative models, existing methods face problems with large irregular masks, often producing blurry textures on the edges of the masked region, semantic inconsistencies, or unconvincing facial structures due to direct pixel level synthesis approach and limited exploitation of facial priors. In this paper we propose a novel architecture, which address these above challenges through semantic-guided hierarchical synthesis. Our approach starts with a method that organizes and synthesizes information based on meaning, followed by refining the texture. This process gives clear insights into the facial structure before we move on to creating detailed images. In the first stage, we blend two techniques: one that focuses on local features with CNNs and global features with Vision Transformers. This helped us create clear and detailed semantic layouts. In the second stage, we use a Multi-Modal Texture Generator to refine these layouts by pulling in information from different scales, ensuring everything looks cohesive and consistent. The architecture naturally handles arbitrary mask configurations through dynamic attention without maskspecific training. Experiment on two datasets CelebA-HQ and FFHQ shows that our model outperforms other state-of-the-art methods, showing improvements in metrics like LPIPS, PSNR, and SSIM. It produces visually striking results with better semantic preservation, in challenging large-area inpainting situations.
Abstract (translated)
面部图像修复（Facial Image Inpainting）的目标是恢复脸部图像中丢失或损坏的区域，同时保持身份识别、结构一致性和逼真的照片质量。这一任务专门用于图片修复领域。尽管近年来深度生成模型取得了显著进展，现有方法在处理大面积不规则遮罩时仍面临挑战，通常会导致边缘模糊、语义不一致性或面部结构不合理等问题，这是因为直接采用像素级合成的方法和有限地利用面部先验知识所致。 本文提出了一种新架构，通过语义引导的层次化综合解决了上述问题。我们的方法首先基于意义组织并合成了信息，然后细化纹理，从而在创造详细图像之前提供了对面部结构清晰的见解。第一阶段中，我们融合了两种技术：一种专注于使用CNN进行局部特征分析，另一种则利用Vision Transformers对全局特征进行处理。这有助于创建清晰且详细的语义布局。 第二阶段中，我们采用多模态纹理生成器来细化这些布局，并从不同尺度获取信息，确保一切看起来连贯一致。该架构通过动态注意力机制自然地处理了任意遮罩配置，无需针对特定遮罩的训练。在CelebA-HQ和FFHQ两个数据集上的实验表明，我们的模型优于其他最先进的方法，在LPIPS、PSNR和SSIM等指标上显示出改进，并在具有挑战性的大面积修复情况下产生了视觉效果更佳且语义保存更好的结果。 简而言之，所提出的新型架构通过引入层次化综合和多模态纹理生成器，显著改善了面部图像的修复质量。这种方法不仅提高了视觉质量和结构一致性，而且对于解决复杂的大面积损坏问题也展现了强大的潜力。
URL
https://arxiv.org/abs/2512.05039
