Abstract
Image inpainting is a fundamental research area between image editing and image generation. Recent state-of-the-art (SOTA) methods have explored novel attention mechanisms, lightweight architectures, and context-aware modeling, demonstrating impressive performance. However, they often struggle with complex structure (e.g., texture, shape, spatial relations) and semantics (e.g., color consistency, object restoration, and logical correctness), leading to artifacts and inappropriate generation. To address this challenge, we design a simple yet effective inpainting paradigm called latent categories guidance, and further propose a diffusion-based model named PixelHacker. Specifically, we first construct a large dataset containing 14 million image-mask pairs by annotating foreground and background (potential 116 and 21 categories, respectively). Then, we encode potential foreground and background representations separately through two fixed-size embeddings, and intermittently inject these features into the denoising process via linear attention. Finally, by pre-training on our dataset and fine-tuning on open-source benchmarks, we obtain PixelHacker. Extensive experiments show that PixelHacker comprehensively outperforms the SOTA on a wide range of datasets (Places2, CelebA-HQ, and FFHQ) and exhibits remarkable consistency in both structure and semantics. Project page at this https URL.
Abstract (translated)
图像修复是图像编辑和图像生成领域中的一个基本研究方向。最近的最先进的（SOTA）方法探索了新颖的注意力机制、轻量级架构以及上下文感知建模，展现了令人印象深刻的表现力。然而，它们在处理复杂结构（例如纹理、形状及空间关系）和语义信息（如颜色一致性、对象恢复及逻辑正确性）时常常遇到困难，导致生成图像中出现伪影或不合适的细节。为解决这一挑战，我们设计了一种简单但有效的修复范式，称为潜在类别引导，并进一步提出了一种基于扩散模型的方法，命名为PixelHacker。 具体而言，我们首先构建了一个包含1400万张图像-掩膜对的大型数据集，通过标注前景和背景（分别为116个和21个潜在类别）来创建。接着，我们分别通过对两个固定大小的嵌入编码潜在前景和背景表示，并在去噪过程中间歇地注入这些特征，使用线性注意力机制完成这一过程。最后，在我们的数据集上进行预训练并在开源基准测试中进行微调之后，得到了PixelHacker模型。 广泛的实验表明，PixelHacker在多个数据集（Places2、CelebA-HQ 和 FFHQ）上的表现全面超越了最先进的方法，并且无论是在结构还是语义方面都展现出了卓越的一致性。项目页面在此 https URL.
URL
https://arxiv.org/abs/2504.20438
