Abstract
Transformers, while powerful, suffer from quadratic computational complexity and the ever-growing Key-Value (KV) cache of the attention mechanism. This paper introduces Trellis, a novel Transformer architecture with bounded memory that learns how to compress its key-value memory dynamically at test time. Trellis replaces the standard KV cache with a fixed-size memory and train a two-pass recurrent compression mechanism to store new keys and values into memory. To achieve this, it leverages an online gradient descent procedure with a forget gate, enabling the compressed memory to be updated recursively while learning to retain important contextual information from incoming tokens at test time. Extensive experiments on language modeling, common-sense reasoning, recall-intensive tasks, and time series show that the proposed architecture outperforms strong baselines. Notably, its performance gains increase as the sequence length grows, highlighting its potential for long-context applications.
Abstract (translated)
尽管Transformer模型非常强大，但它们面临着二次计算复杂度和注意力机制中的键值(KV)缓存不断增长的问题。本文介绍了一种名为Trellis的新颖的Transformer架构，该架构具有有界内存，并且能够在测试时动态地学习如何压缩其键值记忆。Trellis用固定大小的记忆替换了标准KV缓存，并训练了一个两遍递归压缩机制来将新的键和值存储到记忆中。为了实现这一点，它利用了带有遗忘门的在线梯度下降过程，这使得在测试时能够递归地更新压缩后的内存并学习保留来自传入令牌的重要上下文信息。 通过在语言建模、常识推理、需要大量回忆的任务以及时间序列上的广泛实验表明，所提出的架构优于强大的基线模型。值得注意的是，其性能增益随着序列长度的增长而增加，这突显了它在处理长上下文应用方面的潜力。
URL
https://arxiv.org/abs/2512.23852
