Abstract
Precise spatial control in diffusion-based style transfer remains challenging. This challenge arises because diffusion models treat style as a global feature and lack explicit spatial grounding of style representations, making it difficult to restrict style application to specific objects or regions. To our knowledge, existing diffusion models are unable to perform true localized style transfer, typically relying on handcrafted masks or multi-stage post-processing that introduce boundary artifacts and limit generalization. To address this, we propose an attention-supervised diffusion framework that explicitly teaches the model where to apply a given style by aligning the attention scores of style tokens with object masks during training. Two complementary objectives, a Focus loss based on KL divergence and a Cover loss using binary cross-entropy, jointly encourage accurate localization and dense coverage. A modular LoRA-MoE design further enables efficient and scalable multi-style adaptation. To evaluate localized stylization, we introduce the Regional Style Editing Score, which measures Regional Style Matching through CLIP-based similarity within the target region and Identity Preservation via masked LPIPS and pixel-level consistency on unedited areas. Experiments show that our method achieves mask-free, single-object style transfer at inference, producing regionally accurate and visually coherent results that outperform existing diffusion-based editing approaches.
Abstract (translated)
在基于扩散的风格迁移中，实现精确的空间控制仍然是一项挑战。这一难题产生的原因是扩散模型将风格视为全局特征，并且缺乏对风格表示的具体空间定位，这使得限制特定对象或区域内的风格应用变得困难。据我们所知，现有的扩散模型无法执行真正的局部化风格传输，通常依赖于手工设计的掩码或多阶段后处理技术，这些方法会产生边界伪影并限制了泛化能力。 为了解决这个问题，我们提出了一种基于注意力监督的扩散框架。通过在训练过程中将风格标记的注意力分数与对象掩码对齐的方式，该框架明确地教导模型如何应用给定的风格。两个互补的目标——基于KL散度的Focus损失和使用二元交叉熵的Cover损失共同鼓励准确的空间定位和密集覆盖。此外，一个模块化的LoRA-MoE设计进一步实现了高效的多风格适应。 为了评估局部化样式化的效果，我们引入了“区域风格编辑评分”（Regional Style Editing Score），它通过目标区域内的CLIP基相似性测量区域风格匹配，并通过掩码LPIPS和未编辑区域的像素级一致性来衡量身份保持。实验结果表明，我们的方法能够在推理时实现无掩码、单对象的风格转换，生成准确且视觉上一致的结果，优于现有的基于扩散的方法。
URL
https://arxiv.org/abs/2602.19254
