Abstract
How can we generate an image B' that satisfies A:A'::B:B', given the input images A,A' and B? Recent works have tackled this challenge through approaches like visual in-context learning or visual instruction. However, these methods are typically limited to specific models (e.g. InstructPix2Pix. Inpainting models) rather than general diffusion models (e.g. Stable Diffusion, SDXL). This dependency may lead to inherited biases or lower editing capabilities. In this paper, we propose Difference Inversion, a method that isolates only the difference from A and A' and applies it to B to generate a plausible B'. To address model dependency, it is crucial to structure prompts in the form of a "Full Prompt" suitable for input to stable diffusion models, rather than using an "Instruction Prompt". To this end, we accurately extract the Difference between A and A' and combine it with the prompt of B, enabling a plug-and-play application of the difference. To extract a precise difference, we first identify it through 1) Delta Interpolation. Additionally, to ensure accurate training, we propose the 2) Token Consistency Loss and 3) Zero Initialization of Token Embeddings. Our extensive experiments demonstrate that Difference Inversion outperforms existing baselines both quantitatively and qualitatively, indicating its ability to generate more feasible B' in a model-agnostic manner.
Abstract (translated)
如何在给定输入图像A、A'和B的情况下生成一个满足A:A'::B:B'的图像B'? 近期的研究通过视觉上下文学习或视觉指令等方法来解决这一挑战，但这些方法通常局限于特定模型（如InstructPix2Pix、去噪模型）而不是通用扩散模型（如Stable Diffusion、SDXL）。这种依赖性可能导致继承偏见或较低的编辑能力。在本文中，我们提出了差异反转法，这是一种仅从A和A'之间提取差异并将其应用于B以生成合理的B'的方法。为了减少对特定模型的依赖，重要的是将提示结构为适用于稳定扩散模型的“完整提示”，而不是使用“指令提示”。为此，我们准确地提取了A和A'之间的差异，并将其与B的提示相结合，从而实现差异的即插即用应用。 为了精确提取差异，我们首先通过1）Delta插值来识别它。此外，为确保训练准确性，我们提出了2）Token一致性损失和3）Token嵌入零初始化。我们的大量实验表明，差异反转法在定量和定性上都优于现有的基线方法，这表明它可以以模型无关的方式生成更可行的B'。
URL
https://arxiv.org/abs/2506.07750
