Abstract
Positional encodings are essential to transformer-based generative models, yet their behavior in multimodal and attention-sharing settings is not fully understood. In this work, we present a principled analysis of Rotary Positional Embeddings (RoPE), showing that RoPE naturally decomposes into frequency components with distinct positional sensitivities. We demonstrate that this frequency structure explains why shared-attention mechanisms, where a target image is generated while attending to tokens from a reference image, can lead to reference copying, in which the model reproduces content from the reference instead of extracting only its stylistic cues. Our analysis reveals that the high-frequency components of RoPE dominate the attention computation, forcing queries to attend mainly to spatially aligned reference tokens and thereby inducing this unintended copying behavior. Building on these insights, we introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment. Applied to modern transformer-based diffusion architectures, where all tokens share attention, this modulation restores stable and meaningful shared attention. As a result, it enables effective control over the degree of style transfer versus content copying, yielding a proper style-aligned generation process in which stylistic attributes are transferred without duplicating reference content.
Abstract (translated)
位置编码是基于变压器的生成模型中的关键组成部分，然而，在多模态和注意力共享设置下的行为尚未完全理解。在本文中，我们对旋转位置嵌入（RoPE）进行了系统的分析，揭示了RoPE自然地分解成具有不同位置敏感性的频率分量。我们展示了这种频率结构能够解释为什么共用注意力机制——即在关注参考图像中的标记时生成目标图像——会导致复制参考内容的行为，而不是仅仅提取其风格提示。 我们的分析发现，RoPE的高频部分在注意力计算中占主导地位，迫使查询主要关注空间对齐的参考标记，从而导致这种意外的复制行为。基于这些见解，我们提出了一种方法来选择性地调节RoPE频率带，使得注意力反映的是语义相似性而不是严格的位置对齐。当应用于现代基于变压器的扩散架构中所有令牌共用注意力的情况下时，这种调制恢复了稳定且有意义的共享注意力。因此，它能够有效地控制风格转移的程度与内容复制之间的平衡，在此过程中传递风格属性而不重复参考内容，从而实现正确的风格一致生成过程。 这段翻译详细阐述了RoPE在多模态和共享注意力机制中的行为，并提出了一种调节方法以改善模型性能，使之更适于有效的样式迁移而非内容复制。
URL
https://arxiv.org/abs/2602.05013
