Abstract
We provide causal mechanistic validation that in-context learning (ICL) decomposes into two separable mechanisms: Task Schema (abstract task type recognition) and Binding (specific input-output associations). Through activation patching experiments across 9 models from 7 Transformer families plus Mamba (370M-13B parameters), we establish three key findings: 1. Double dissociation: Task Schema transfers at 100% via late MLP patching; Binding transfers at 62% via residual stream patching -- proving separable mechanisms 2. Prior-Schema trade-off: Schema reliance inversely correlates with prior knowledge (Spearman rho = -0.596, p < 0.001, N=28 task-model pairs) 3. Architecture generality: The mechanism operates across all tested architectures including the non-Transformer Mamba These findings offer a mechanistic account of the ICL puzzle that contrasts with prior views treating ICL as a monolithic mechanism (whether retrieval-based, gradient descent-like, or purely Bayesian). By establishing that Schema and Binding are neurally dissociable -- not merely behavioral modes -- we provide causal evidence for dual-process theories of ICL. Models rely on Task Schema when prior knowledge is absent, but prior knowledge interferes through attentional mis-routing (72.7% recency bias) rather than direct output competition (0%). This explains why arbitrary mappings succeed (zero prior leads to full Schema reliance) while factual overrides fail -- and reveals that the true bottleneck is attentional, not output-level. Practical implications: Understanding these dual mechanisms enables more efficient prompt engineering -- reliable schema transfer reduces required demonstrations for novel tasks, while prior-aware design can mitigate the 38% binding failure rate in high-prior scenarios, improving ICL system reliability in production deployments.
Abstract (translated)
我们提供了关于基于上下文学习（ICL）的因果机制验证，证明ICL可以分解为两个可分离的机制：任务模式识别（Task Schema）和绑定（Binding特定输入-输出关联）。通过在来自7个Transformer家族以及Mamba模型系列中的9种不同架构（参数量从370M到13B不等）上进行激活补丁实验，我们得出了三个关键发现： 1. **双重分离**：任务模式识别可以通过对后期MLP层的补丁完全转移（100%），而绑定机制通过残差流补丁可以以62%的效率进行转移——这证明了它们是可分的机制。 2. **先验-模式权衡**：模式依赖性与先前知识存在反向相关关系（Spearman rho = -0.596，p < 0.001，N=28任务模型对），这意味着随着先前知识的增加，对于特定任务的任务模式识别的重要性会降低。 3. **架构通用性**：这种机制在所有测试过的架构中运行良好，包括非Transformer架构Mamba。 这些发现为ICL谜题提供了一个机械解释，并与之前将ICL视为单一机制（无论是基于检索、类似梯度下降还是纯粹贝叶斯）的观点相区别。通过证实模式和绑定从神经元层面上可以分离——不仅仅是行为模式的不同——我们为双过程理论提供了因果证据。 模型在缺乏先前知识时依赖于任务模式识别，但是当存在先前知识时会干扰注意力的正确路由（72.7%近期偏差），而不是直接输出竞争（0%）。这解释了为什么任意映射能够成功（零先验导致完全模式依赖）而事实覆盖失败，并揭示了真正的瓶颈在于注意层面而非输出层级。 **实用意义**：理解这些双机制可以更高效地进行提示工程——可靠的任务模式转移减少了对于新任务演示的需求，同时基于先前知识的设计可以在高先验场景中降低38%的绑定失败率，从而提高ICL系统在实际部署中的可靠性。
URL
https://arxiv.org/abs/2512.17325
