Abstract
Match-and-copy is a core retrieval primitive used at inference time by large language models to retrieve a matching token from the context then copy its successor. Yet, understanding how this behavior emerges on natural data is challenging because retrieval and memorization are entangled. To disentangle the two, we introduce Gaussian Match-and-Copy (GMC), a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals. Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits in practice, and separates architectures by their retrieval capabilities. We also analyze the optimization dynamics in a simplified attention setting. Although many solutions are a priori possible under a regression objective, including ones that do not implement retrieval, we identify an implicit-bias regime in which gradient descent drives the parameters to diverge while their direction aligns with the max-margin separator, yielding hard match selection. We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.
Abstract (translated)
匹配与复制（Match-and-Copy）是大型语言模型在推理过程中用于从上下文中检索匹配的标记然后复制其后继标记的核心检索机制。然而，由于检索和记忆之间的纠缠关系，在自然数据上理解这种行为的出现具有挑战性。为了分离这两种现象，我们引入了高斯匹配与复制（GMC），这是一个极简主义基准测试，通过纯粹的二阶相关信号来隔离长距离检索。 数值研究显示，这个任务保留了Transformer在实践中发展匹配与复制电路的关键定性方面，并通过其检索能力将不同架构区分开。我们还在简化注意力设置中分析了优化动力学。尽管根据回归目标有许多可能的解决方案（包括不实施检索的方法），但我们确定了一个隐含偏置（implicit bias）的区域，在此区域内，梯度下降会驱使参数发散的同时让它们的方向与最大间隔分割器对齐，从而实现硬匹配选择。我们证明了在明确的技术条件下，当GD轨迹达到零经验损失时，会发生这种最大间隔对齐。 简而言之，这项研究通过引入GMC基准测试和分析优化动力学来探究大型语言模型中的检索机制，并揭示了梯度下降如何促进特定的检索行为。
URL
https://arxiv.org/abs/2602.07562
