Abstract
Text-guided image inpainting aims to inpaint masked image regions based on a textual prompt while preserving the background. Although diffusion-based methods have become dominant, their property of modeling the entire image in latent space makes it challenging for the results to align well with prompt details and maintain a consistent background. To address these issues, we explore Mask AutoRegressive (MAR) models for this task. MAR naturally supports image inpainting by generating latent tokens corresponding to mask regions, enabling better local controllability without altering the background. However, directly applying MAR to this task makes the inpainting content either ignore the prompts or be disharmonious with the background context. Through analysis of the attention maps from the inpainting images, we identify the impact of background tokens on text tokens during the MAR generation, and leverage this to design \textbf{Token Painter}, a training-free text-guided image inpainting method based on MAR. Our approach introduces two key components: (1) Dual-Stream Encoder Information Fusion (DEIF), which fuses the semantic and context information from text and background in frequency domain to produce novel guidance tokens, allowing MAR to generate text-faithful inpainting content while keeping harmonious with background context. (2) Adaptive Decoder Attention Score Enhancing (ADAE), which adaptively enhances attention scores on guidance tokens and inpainting tokens to further enhance the alignment of prompt details and the content visual quality. Extensive experiments demonstrate that our training-free method outperforms prior state-of-the-art methods across almost all metrics and delivers superior visual results. Codes will be released.
Abstract (translated)
文本引导的图像修复（Text-guided image inpainting）旨在根据给定的文字提示来修补被遮罩的部分，同时保持背景不变。尽管基于扩散的方法已经变得非常流行，但其在潜在空间中模拟整个图像的特点使得结果难以与文字描述细节很好地对齐，并且也很难维持背景的一致性。为了解决这些问题，我们探索了使用Mask AutoRegressive (MAR) 模型来进行这项任务的可能性。MAR模型通过生成对应于掩码区域的潜在令牌来自然地支持图像修复工作，这使得在不改变背景的情况下实现更好的局部可控性成为可能。然而，直接将MAR应用于该任务会导致修复内容要么忽视提示信息，要么与背景环境不协调。 通过对修复后的图像中注意力图的分析，我们发现背景令牌对文本令牌的影响，并利用这一点设计了\textbf{Token Painter}——一种基于MAR的、无需训练即可实现文本引导图像修补的方法。我们的方法引入了两个关键组件： 1. **双流编码器信息融合（Dual-Stream Encoder Information Fusion, DEIF）**：该机制在频域中融合来自文字和背景的信息，生成新的指导令牌，使MAR能够产生与提示一致的修复内容，并且保持与背景环境的一致性。 2. **自适应解码器注意力评分增强（Adaptive Decoder Attention Score Enhancing, ADAE）**：通过自适应地提高指导令牌和修补令牌上的注意分数来进一步加强提示细节与内容视觉质量之间的对齐。 广泛的实验表明，我们提出的无需训练的方法在几乎所有指标上都优于现有的最先进的方法，并且提供了更优秀的视觉结果。代码将公开发布。
URL
https://arxiv.org/abs/2509.23919
