Abstract
Despite the remarkable practical success of transformer-based language models, recent work has raised concerns about their ability to perform state tracking. In particular, a growing body of literature has shown this limitation primarily through failures in out-of-distribution (OOD) generalization, such as length extrapolation. In this work, we shift attention to the in-distribution implications of these limitations. We conduct a large-scale experimental study of the data efficiency of transformers and recurrent neural networks (RNNs) across multiple supervision regimes. We find that the amount of training data required by transformers grows much more rapidly with state-space size and sequence length than for RNNs. Furthermore, we analyze the extent to which learned state-tracking mechanisms are shared across different sequence lengths. We show that transformers exhibit negligible or even detrimental weight sharing across lengths, indicating that they learn length-specific solutions in isolation. In contrast, recurrent models exhibit effective amortized learning by sharing weights across lengths, allowing data from one sequence length to improve performance on others. Together, these results demonstrate that state tracking remains a fundamental challenge for transformers, even when training and evaluation distributions match.
Abstract (translated)
尽管基于变压器的语言模型在实践中取得了显著的成功，但最近的研究对它们执行状态跟踪的能力提出了担忧。特别是，越来越多的文献通过分布外（OOD）泛化中的失败来展示了这种限制，例如长度推断。在这项工作中，我们将注意力转向这些限制在分布内的影响。我们进行了大规模实验研究，比较了变压器和循环神经网络（RNNs）在不同监督模式下的数据效率。我们发现，随着状态空间大小和序列长度的增加，变压器所需的数据量增长得比RNN要快得多。 此外，我们还分析了学习到的状态跟踪机制在不同序列长度之间共享的程度。结果显示，变压器在不同长度之间的权重共享非常有限或甚至是有害的，这表明它们是孤立地学习特定于长度的解决方案。相比之下，循环模型通过跨长度共享权重来实现有效的样本外推学习，使得一个序列长度的数据可以改善其他长度序列的表现。 综合这些结果表明，即使训练和评估分布相匹配的情况下，状态跟踪依然是变压器面临的一个基本挑战。
URL
https://arxiv.org/abs/2602.18333
