Abstract
This paper presents a direct framework for sequence models with hidden states on closed subgroups of U(d). We use a minimal axiomatic setup and derive recurrent and transformer templates from a shared skeleton in which subgroup choice acts as a drop-in replacement for state space, tangent projection, and update map. We then specialize to O(d) and evaluate orthogonal-state RNN and transformer models on Tiny Shakespeare and Penn Treebank under parameter-matched settings. We also report a general linear-mixing extension in tangent space, which applies across subgroup choices and improves finite-budget performance in the current O(d) experiments.
Abstract (translated)
本文提出了一种直接框架，用于在U(d)的闭子群上的隐状态序列模型。我们使用了最小公理设置，并从一个共享骨架中推导出了递归和变换模板，在此骨架中，子群选择作为状态空间、切线投影和更新映射的即插即用替换。然后，我们将框架应用于O(d)，并在参数匹配设置下对Tiny Shakespeare和Penn Treebank数据集进行了正交状态RNN和Transformer模型的评估。此外，我们还报告了一种通用的线性混合扩展方法，在切空间中应用，并且这种扩展适用于各种子群选择，可改进当前O(d)实验中的有限预算性能。
URL
https://arxiv.org/abs/2602.18417
