Abstract
Recent advances in diffusion models have enhanced multimodal-guided visual generation, enabling customized subject insertion that seamlessly "brushes" user-specified objects into a given image guided by textual prompts. However, existing methods often struggle to insert customized subjects with high fidelity and align results with the user's intent through textual prompts. In this work, we propose "In-Context Brush", a zero-shot framework for customized subject insertion by reformulating the task within the paradigm of in-context learning. Without loss of generality, we formulate the object image and the textual prompts as cross-modal demonstrations, and the target image with the masked region as the query. The goal is to inpaint the target image with the subject aligning textual prompts without model tuning. Building upon a pretrained MMDiT-based inpainting network, we perform test-time enhancement via dual-level latent space manipulation: intra-head "latent feature shifting" within each attention head that dynamically shifts attention outputs to reflect the desired subject semantics and inter-head "attention reweighting" across different heads that amplifies prompt controllability through differential attention prioritization. Extensive experiments and applications demonstrate that our approach achieves superior identity preservation, text alignment, and image quality compared to existing state-of-the-art methods, without requiring dedicated training or additional data collection.
Abstract (translated)
最近的扩散模型在多模态引导视觉生成方面的进步，增强了定制对象插入功能，使得用户可以通过文本提示将特定物体无缝“刷入”给定图像。然而，现有的方法通常难以以高保真度插入定制对象，并且很难通过文本提示使结果与用户的意图对齐。为此，我们提出了“In-Context Brush”，这是一种零样本框架，用于基于上下文学习范式重新定义任务的定制对象插入。 具体来说，在不失一般性的前提下，我们将目标图像及其掩码区域作为查询（query），将对象图像和文本提示视为跨模态演示（cross-modal demonstrations）。我们的目标是在不调整模型的情况下，使用这些跨模态信息来填充目标图像中的空缺部分，并确保生成的对象与文本提示对齐。 基于预训练的MMDiT基础填补网络，我们通过双层隐空间操作在测试时间进行增强：在同一注意头内的“隐特征偏移”（latent feature shifting），动态调整注意力输出以反映所需对象语义；以及跨不同头部间的“注意重新加权”（attention reweighting），通过差异化注意力优先级来放大提示控制能力。 广泛的实验和应用表明，与现有最先进的方法相比，我们的方法在身份保留、文本对齐和图像质量方面均表现更优，并且无需专门训练或额外的数据收集。
URL
https://arxiv.org/abs/2505.20271
