Abstract
Attention-based arbitrary style transfer methods, including CNN-based, Transformer-based, and Diffusion-based, have flourished and produced high-quality stylized images. However, they perform poorly on the content and style images with the same semantics, i.e., the style of the corresponding semantic region of the generated stylized image is inconsistent with that of the style image. We argue that the root cause lies in their failure to consider the relationship between local regions and semantic regions. To address this issue, we propose a plug-and-play semantic continuous-sparse attention, dubbed SCSA, for arbitrary semantic style transfer -- each query point considers certain key points in the corresponding semantic region. Specifically, semantic continuous attention ensures each query point fully attends to all the continuous key points in the same semantic region that reflect the overall style characteristics of that region; Semantic sparse attention allows each query point to focus on the most similar sparse key point in the same semantic region that exhibits the specific stylistic texture of that region. By combining the two modules, the resulting SCSA aligns the overall style of the corresponding semantic regions while transferring the vivid textures of these regions. Qualitative and quantitative results prove that SCSA enables attention-based arbitrary style transfer methods to produce high-quality semantic stylized images.
Abstract (translated)
基于注意力的任意风格转换方法，包括CNN（卷积神经网络）基础、Transformer基础和扩散模型基础的方法已经蓬勃发展，并生成了高质量的艺术化图像。然而，当处理内容图与样式图具有相同语义的情况时，这些方法表现不佳，即所生成的艺术化图像中对应语义区域的风格与其参考样式的风格不一致。我们认为根本原因在于它们未能考虑局部区域和语义区域之间的关系。为解决这一问题，我们提出了一种插件式连续稀疏注意力机制（SCSA），专门用于任意语义风格转换——每个查询点都会考虑到其对应语义区域内的特定关键点。 具体而言，连续的语义注意确保了每个查询点能够全面关注同语义区域内反映整体风格特征的所有连续关键点；而稀疏的语义注意则使每个查询点能聚焦于在相同语义区域中展示该区域独特风格纹理的最相似的那个单一关键点。通过结合这两个模块，SCSA能够在进行风格转换时同时对齐对应语义区域的整体风格并转移这些区域的具体纹理细节。 定性和定量结果证明了SCSA能使基于注意力机制的方法产生高质量的语义艺术化图像，并解决了当内容和样式图具有相同语义时生成的艺术化图像中语义区域间风格不一致的问题。
URL
https://arxiv.org/abs/2503.04119
