Abstract
Image fusion seeks to seamlessly integrate foreground objects with background scenes, producing realistic and harmonious fused images. Unlike existing methods that directly insert objects into the background, adaptive and interactive fusion remains a challenging yet appealing task. It requires the foreground to adjust or interact with the background context, enabling more coherent integration. To address this, we propose an iterative human-in-the-loop data generation pipeline, which leverages limited initial data with diverse textual prompts to generate fusion datasets across various scenarios and interactions, including placement, holding, wearing, and style transfer. Building on this, we introduce DreamFuse, a novel approach based on the Diffusion Transformer (DiT) model, to generate consistent and harmonious fused images with both foreground and background information. DreamFuse employs a Positional Affine mechanism to inject the size and position of the foreground into the background, enabling effective foreground-background interaction through shared attention. Furthermore, we apply Localized Direct Preference Optimization guided by human feedback to refine DreamFuse, enhancing background consistency and foreground harmony. DreamFuse achieves harmonious fusion while generalizing to text-driven attribute editing of the fused results. Experimental results demonstrate that our method outperforms state-of-the-art approaches across multiple metrics.
Abstract (translated)
图像融合的目标是无缝地将前景对象与背景场景结合，生成逼真且和谐的合成图像。不同于现有的直接将对象插入到背景中的方法，适应性和交互式的融合仍然是一项具有挑战性但吸引人的任务。这一过程要求前景能够根据背景环境进行调整或互动，从而实现更连贯的整合。为此，我们提出了一种迭代的人机协作数据生成流水线，该流水线利用有限的初始数据和多样化的文本提示来生成涵盖不同场景和交互方式（如放置、手持、穿戴及风格转换）的融合数据集。在此基础上，我们引入了DreamFuse这一基于扩散变换器(DiT)模型的新方法，它能够结合前景与背景信息，生成一致且和谐的合成图像。DreamFuse采用位置归一化机制将前景对象的位置和大小注入到背景中，通过共享注意力实现有效的前、后景互动。此外，我们还应用了由人类反馈引导的局部直接偏好优化技术来改进DreamFuse，增强其在背景一致性及前景和谐性方面的表现。DreamFuse不仅实现了无缝融合，还能根据文本驱动的属性编辑需求进行泛化。实验结果表明，在多个评价指标上我们的方法超越了现有的最先进技术。
URL
https://arxiv.org/abs/2504.08291
