Abstract
Transformers have been established as the de-facto backbones for most recent advances in sequence modeling, mainly due to their growing memory capacity that scales with the context length. While plausible for retrieval tasks, it causes quadratic complexity and so has motivated recent studies to explore viable subquadratic recurrent alternatives. Despite showing promising preliminary results in diverse domains, such recurrent architectures underperform Transformers in recall-intensive tasks, often attributed to their fixed-size memory. In this paper, we introduce Memory Caching (MC), a simple yet effective technique that enhances recurrent models by caching checkpoints of their memory states (a.k.a. hidden states). Memory Caching allows the effective memory capacity of RNNs to grow with sequence length, offering a flexible trade-off that interpolates between the fixed memory (i.e., $O(L)$ complexity) of RNNs and the growing memory (i.e., $O(L^2)$ complexity) of Transformers. We propose four variants of MC, including gated aggregation and sparse selective mechanisms, and discuss their implications on both linear and deep memory modules. Our experimental results on language modeling, and long-context understanding tasks show that MC enhances the performance of recurrent models, supporting its effectiveness. The results of in-context recall tasks indicate that while Transformers achieve the best accuracy, our MC variants show competitive performance, close the gap with Transformers, and performs better than state-of-the-art recurrent models.
Abstract (translated)
Transformer 模型已被确立为大多数最近序列建模进展的核心架构，主要归功于其随上下文长度增加而扩大的内存容量。虽然对于检索任务来说是合理的，但对于 Transformer 来说，这种增长导致了二次复杂度的问题，并促使近期研究探索可行的亚二次递归替代方案。尽管在多个领域展示了初步的有希望的结果，这些递归架构在依赖于高召回的任务中表现不如 Transformer，这通常被归因于其固定的内存大小。 在本文中，我们介绍了一种简单却有效的方法——Memory Caching（MC），通过缓存递归模型的记忆状态检查点来增强递归模型。这种方法使得 RNN 的有效记忆容量随着序列长度的增长而增长，并提供了一个灵活的折衷方案，在固定大小的内存复杂度 ($O(L)$) 和不断扩大的 Transformer 内存复杂度 ($O(L^2)$) 之间进行平衡。 我们提出了四种 MC 变体，包括门控聚合和稀疏选择机制，并讨论了它们在线性和深度记忆模块中的影响。我们在语言建模和长上下文理解任务上的实验结果表明，MC 技术增强了递归模型的表现力和支持其有效性。针对上下文中回忆的任务的结果显示，虽然 Transformer 在准确性方面表现出最佳性能，但我们的 MC 变体展示了具有竞争力的性能，缩小了与 Transformer 的差距，并优于最先进的递归模型。
URL
https://arxiv.org/abs/2602.24281
