Abstract
We prove that a minimal Transformer architecture with frozen weights is capable of emulating a broad class of algorithms by in-context prompting. In particular, for any algorithm implementable by a fixed-weight attention head (e.g. one-step gradient descent or linear/ridge regression), there exists a prompt that drives a two-layer softmax attention module to reproduce the algorithm's output with arbitrary precision. This guarantee extends even to a single-head attention layer (using longer prompts if necessary), achieving architectural minimality. Our key idea is to construct prompts that encode an algorithm's parameters into token representations, creating sharp dot-product gaps that force the softmax attention to follow the intended computation. This construction requires no feed-forward layers and no parameter updates. All adaptation happens through the prompt alone. These findings forge a direct link between in-context learning and algorithmic emulation, and offer a simple mechanism for large Transformers to serve as prompt-programmable libraries of algorithms. They illuminate how GPT-style foundation models may swap algorithms via prompts alone, establishing a form of algorithmic universality in modern Transformer models.
Abstract (translated)
我们证明，一个权重固定化的最小Transformer架构能够通过上下文提示来模拟一大类算法。具体来说，对于任何可以通过固定权重的注意力头实现的算法（例如一步梯度下降或线性/岭回归），都存在一种提示，可以驱动两层softmax注意力模块以任意精度重现该算法的输出。这一保证甚至适用于单个头部注意层（如果需要的话使用更长的提示），实现了架构上的最小化。我们的核心思想是构造能够将算法参数编码为标记表示形式的提示，创造出明确的点积差距，迫使softmax注意机制按照预期进行计算。这种构建不需要前向传播层和参数更新。所有适应性都通过提示单独完成。 这些发现直接建立了上下文学习与算法模拟之间的联系，并提供了一种简单的机制，使大型Transformer可以充当可通过提示编程的算法库。它们揭示了GPT风格的基础模型如何仅通过提示即可交换算法，从而在现代Transformer模型中建立了一种形式的算法普适性。
URL
https://arxiv.org/abs/2508.17550
