Abstract
Diffusion models have demonstrated remarkable performance in image generation, particularly within the domain of style transfer. Prevailing style transfer approaches typically leverage pre-trained diffusion models' robust feature extraction capabilities alongside external modular control pathways to explicitly impose style guidance signals. However, these methods often fail to capture complex style reference or retain the identity of user-provided content images, thus falling into the trap of style-content balance. Thus, we propose a training-free style transfer approach via $\textbf{h}$eterogeneous $\textbf{a}$ttention $\textbf{m}$odulation ($\textbf{HAM}$) to protect identity information during image/text-guided style reference transfer, thereby addressing the style-content trade-off challenge. Specifically, we first introduces style noise initialization to initialize latent noise for diffusion. Then, during the diffusion process, it innovatively employs HAM for different attention mechanisms, including Global Attention Regulation (GAR) and Local Attention Transplantation (LAT), which better preserving the details of the content image while capturing complex style references. Our approach is validated through a series of qualitative and quantitative experiments, achieving state-of-the-art performance on multiple quantitative metrics.
Abstract (translated)
扩散模型在图像生成领域展现出卓越性能，尤其在风格迁移任务中表现突出。现有风格迁移方法通常利用预训练扩散模型强大的特征提取能力，结合外部模块化控制路径来显式施加风格引导信号。然而，这些方法往往难以捕捉复杂风格参考或保留用户提供内容图像的身份特征，从而陷入风格-内容平衡的困境。因此，我们提出一种无需训练的异质注意力调制（HAM）风格迁移方法，以在图像/文本引导的风格参考迁移过程中保护身份信息，从而解决风格-内容权衡挑战。具体而言，我们首先引入风格噪声初始化来为扩散过程初始化潜在噪声。随后在扩散过程中，创新性地对不同的注意力机制采用HAM，包括全局注意力调控（GAR）与局部注意力移植（LAT），在捕捉复杂风格参考的同时更好地保留内容图像的细节。通过一系列定性与定量实验验证了本方法，在多项定量指标上达到了最先进性能。
URL
https://arxiv.org/abs/2603.24043
