Abstract
This paper introduces the Polynomial Mixer (PoM), a novel token mixing mechanism with linear complexity that serves as a drop-in replacement for self-attention. PoM aggregates input tokens into a compact representation through a learned polynomial function, from which each token retrieves contextual information. We prove that PoM satisfies the contextual mapping property, ensuring that transformers equipped with PoM remain universal sequence-to-sequence approximators. We replace standard self-attention with PoM across five diverse domains: text generation, handwritten text recognition, image generation, 3D modeling, and Earth observation. PoM matches the performance of attention-based models while drastically reducing computational cost when working with long sequences. The code is available at this https URL.
Abstract (translated)
本文介绍了多项式混合器（PoM），这是一种具有线性复杂度的新型令牌混合机制，可作为自注意力机制的即插式替代方案。PoM通过一个学习到的多项式函数将输入令牌聚合为紧凑表示，每个令牌再从中检索上下文信息。我们证明PoM满足上下文映射属性，确保配备PoM的变换器仍保持通用序列到序列逼近能力。我们在五个不同领域用PoM替换了标准自注意力机制：文本生成、手写文本识别、图像生成、3D建模和地球观测。PoM在长序列处理中匹配了基于注意力模型的性能，同时大幅降低了计算成本。代码已公开于该链接。
URL
https://arxiv.org/abs/2604.06129
