Abstract
Text-guided semantic manipulation refers to semantically editing an image generated from a source prompt to match a target prompt, enabling the desired semantic changes (e.g., addition, removal, and style transfer) while preserving irrelevant contents. With the powerful generative capabilities of the diffusion model, the task has shown the potential to generate high-fidelity visual content. Nevertheless, existing methods either typically require time-consuming fine-tuning (inefficient), fail to accomplish multiple semantic manipulations (poorly extensible), and/or lack support for different modality tasks (limited generalizability). Upon further investigation, we find that the geometric properties of noises in the diffusion model are strongly correlated with the semantic changes. Motivated by this, we propose a novel $\textit{GTF}$ for text-guided semantic manipulation, which has the following attractive capabilities: 1) $\textbf{Generalized}$: our $\textit{GTF}$ supports multiple semantic manipulations (e.g., addition, removal, and style transfer) and can be seamlessly integrated into all diffusion-based methods (i.e., Plug-and-play) across different modalities (i.e., modality-agnostic); and 2) $\textbf{Training-free}$: $\textit{GTF}$ produces high-fidelity results via simply controlling the geometric relationship between noises without tuning or optimization. Our extensive experiments demonstrate the efficacy of our approach, highlighting its potential to advance the state-of-the-art in semantics manipulation.
Abstract (translated)
文本引导的语义编辑是指在基于源提示生成图像后，通过对该图像进行语义上的修改使其与目标描述相匹配的过程。这个过程可以实现诸如添加、移除或风格转换等所需的语义变化，同时保持无关内容不变。借助扩散模型强大的生成能力，这项任务已经显示出能够生成高保真视觉内容的潜力。然而，现有的方法要么需要耗时的微调（效率低下），无法完成多样的语义编辑操作（扩展性差），或者不支持不同模态的任务（泛化能力弱）。进一步研究发现，扩散模型中噪声的几何属性与语义变化之间有很强的相关性。受此启发，我们提出了一种新颖的方法——$\textit{GTF}$ (Generalized Training-free Framework)，用于文本引导的语义编辑，该方法具备以下优点： 1. **通用化**：我们的 $\textit{GTF}$ 支持多种语义操作（如添加、移除和风格转换），并且可以无缝地集成到所有基于扩散模型的方法中，在不同模态下均能使用（即插即用）。 2. **无需训练**：$\textit{GTF}$ 通过简单控制噪声之间的几何关系即可生成高保真的结果，而不需要任何微调或优化步骤。 我们的大量实验展示了这种方法的有效性，并突显了其在语义编辑领域中的领先地位和应用潜力。
URL
https://arxiv.org/abs/2504.17269
