Abstract
Style transfer in diffusion models enables controllable visual generation by injecting the style of a reference image. However, recent encoder-based methods, while efficient and tuning-free, often suffer from content leakage, where semantic elements from the style image undesirably appear in the output, impairing prompt fidelity and stylistic consistency. In this work, we introduce CleanStyle, a plug-and-play framework that filters out content-related noise from the style embedding without retraining. Motivated by empirical analysis, we observe that such leakage predominantly stems from the tail components of the style embedding, which are isolated via Singular Value Decomposition (SVD). To address this, we propose CleanStyleSVD (CS-SVD), which dynamically suppresses tail components using a time-aware exponential schedule, providing clean, style-preserving conditional embeddings throughout the denoising process. Furthermore, we present Style-Specific Classifier-Free Guidance (SS-CFG), which reuses the suppressed tail components to construct style-aware unconditional inputs. Unlike conventional methods that use generic negative embeddings (e.g., zero vectors), SS-CFG introduces targeted negative signals that reflect style-specific but prompt-irrelevant visual elements. This enables the model to effectively suppress these distracting patterns during generation, thereby improving prompt fidelity and enhancing the overall visual quality of stylized outputs. Our approach is lightweight, interpretable, and can be seamlessly integrated into existing encoder-based diffusion models without retraining. Extensive experiments demonstrate that CleanStyle substantially reduces content leakage, improves stylization quality and improves prompt alignment across a wide range of style references and prompts.
Abstract (translated)
在扩散模型中，风格转换技术通过注入参考图像的风格实现可控的视觉生成。然而，最近基于编码器的方法尽管高效且无需调整参数，但常常会遭受内容渗漏的问题：即从风格图提取出的一些语义元素不希望地出现在输出结果中，从而损害了提示准确性和风格一致性。为此，在这项工作中我们引入了一种名为CleanStyle的插件式框架，该框架能够在无需重新训练的情况下过滤掉与内容相关的噪声。 基于实证分析，我们观察到这种渗漏主要来源于样式嵌入中的尾部成分（通过奇异值分解SVD来隔离）。为解决这一问题，我们提出了CleanStyleSVD (CS-SVD)，一种使用时间感知指数调度动态抑制尾部组件的方法。这在整个去噪过程中提供了既干净又保持风格的条件嵌入。 此外，我们还提出了一种风格特定的无分类器引导（Style-Specific Classifier-Free Guidance, SS-CFG）方法，该方法重用被压制的尾部成分来构建具有感知到的风格但与提示无关的无条件输入。不同于传统的方法使用通用负向嵌入（如零向量），SS-CFG引入了针对特定风格但与提示无关的视觉元素的目标化负面信号。这使模型能够在生成过程中有效抑制这些干扰模式，从而提高了提示准确性并提升了整体视觉质量。 我们的方法重量级轻、解释性强，并且可以无缝地集成到现有的基于编码器的扩散模型中而无需重新训练。大量的实验表明，CleanStyle大幅减少了内容渗漏，改进了风格化质量，增强了在广泛风格参考和提示下的指令对齐度。
URL
https://arxiv.org/abs/2602.20721
