Abstract
We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning? We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants. To explore an alternative, we propose an attention-free architecture based on Grassmann flows. Instead of forming an L by L attention matrix, our Causal Grassmann layer (i) linearly reduces token states, (ii) encodes local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates, and (iii) fuses these geometric features back into the hidden states through gated mixing. Information therefore propagates by controlled deformations of low-rank subspaces over multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space. On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT slightly outperforms a Transformer head, with best validation and test accuracies of 0.8550 and 0.8538 compared to 0.8545 and 0.8511. We analyze the complexity of Grassmann mixing, show linear scaling in sequence length for fixed rank, and argue that such manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.
Abstract (translated)
我们重新审视了序列建模中的一个基本问题：显式的自注意力机制对于实现优异性能和推理而言是否真的必不可少？我们认为，标准的多头注意力机制最好被视为一种张量提升形式：隐藏向量被映射到一个高维空间，在这个空间中包含成对交互，并且通过梯度下降约束这种提升后的张量来进行学习。虽然这种机制非常具有表达能力，但由于在经过许多层后变得难以用一组显式的不变性来描述，因此从数学上讲它相当不透明。 为了探索一种替代方案，我们提出了一种基于Grassmann流的无注意力架构。与形成一个L×L的注意力矩阵不同，我们的因果Grassmann层（i）线性地减少令牌状态，（ii）通过Plücker坐标将局部令牌对编码为Grassmann流形上的二维子空间，并且（iii）通过门控混合将这些几何特征重新融合回隐藏状态。因此信息传播是通过对多尺度局部窗口中的低秩子空间进行受控变形来实现的，核心计算发生在有限维流形上而非无结构张量空间内。 在Wikitext-2语言建模基准测试中，仅基于Grassmann的模型（具有13至18百万参数）达到了约比同规模Transformer小10%到15%的有效困惑度。对于SNLI自然语言推理任务，在DistilBERT之上添加了一个Grassmann-Plücker头部，在验证和测试准确性上略微优于一个转换器头部，最佳验证与测试准确率分别为0.8550和0.8538，相比之下，后者为0.8545和0.8511。 我们分析了Grassmann混合的复杂性，并展示了对于固定秩，在序列长度上的线性缩放。我们认为基于流形的设计提供了一条更结构化的路径，能够朝着几何及不变性解释神经推理方向前进。
URL
https://arxiv.org/abs/2512.19428
