Abstract
Transformers require positional encodings to represent sequence order, yet most prior work focuses on designing new positional encodings rather than examining how positional information is fused with token embeddings. In this paper, we study whether the fusion mechanism itself affects performance, particularly in long-sequence settings. We conduct a controlled empirical study comparing three canonical fusion strategies--element-wise addition, concatenation with projection, and scalar gated fusion--under identical Transformer architectures, data splits, and random seeds. Experiments on three text classification datasets spanning short (AG News), medium (IMDB), and long (ArXiv) sequences show that fusion choice has negligible impact on short texts but produces consistent gains on long documents. To verify that these gains are structural rather than stochastic, we perform paired-seed analysis and cross-dataset comparison across sequence-length regimes. Additional experiments on the ArXiv dataset indicate that the benefit of learnable fusion generalizes across multiple positional encoding families. Finally, we explore a lightweight convolutional gating mechanism that introduces local inductive bias at the fusion level, evaluated on long documents only. Our results indicate that positional-encoding fusion is a non-trivial design choice for long-sequence Transformers and should be treated as an explicit modeling decision rather than a fixed default.
Abstract (translated)
翻译如下： 转换器（Transformers）需要位置编码来表示序列顺序，然而大多数先前的工作主要集中在设计新的位置编码上，而不是研究如何将位置信息与标记嵌入融合。在这篇论文中，我们探讨了融合机制本身是否会影响性能，特别是在长序列设置下。我们在相同的Transformer架构、数据分割和随机种子条件下，对三种标准的融合策略——元素级相加、连接后投影以及标量门控融合进行了有控制的经验研究。在三个文本分类数据集（包括短文AG News、中等长度IMDB和长文档ArXiv）上的实验表明，在短文中选择融合方式几乎不会影响性能，但在长文档上则会产生一致的改进效果。为了验证这些改进是结构性而非随机性的，我们进行了配对种子分析及跨序列长度范围的数据集比较。在ArXiv数据集上的额外实验证明了可学习的融合方法能够跨多种位置编码家族进行泛化。最后，我们探索了一种轻量级的卷积门控机制，在长文档上对其进行评估，以引入局部归纳偏差。我们的结果表明，对于长序列转换器而言，位置编码的融合是一个重要的设计选择，并且应该被视为一个明确的建模决策而非固定的默认选项。
URL
https://arxiv.org/abs/2601.05807
