Abstract
In-context learning (ICL) has garnered significant attention for its ability to grasp functions/tasks from demonstrations. Recent studies suggest the presence of a latent task/function vector in LLMs during ICL. Merullo et al. (2024) showed that LLMs leverage this vector alongside the residual stream for Word2Vec-like vector arithmetic, solving factual-recall ICL tasks. Additionally, recent work empirically highlighted the key role of Question-Answer data in enhancing factual-recall capabilities. Despite these insights, a theoretical explanation remains elusive. To move one step forward, we propose a theoretical framework building on empirically grounded hierarchical concept modeling. We develop an optimization theory, showing how nonlinear residual transformers trained via gradient descent on cross-entropy loss perform factual-recall ICL tasks via vector arithmetic. We prove 0-1 loss convergence and show the strong generalization, including robustness to concept recombination and distribution shifts. These results elucidate the advantages of transformers over static embedding predecessors. Empirical simulations corroborate our theoretical insights.
Abstract (translated)
在上下文学习（ICL）中，模型从演示中掌握功能/任务的能力引起了广泛关注。近期研究表明，在进行ICL时，大型语言模型（LLM）内部存在一个潜在的任务/函数向量。Merullo等人(2024)发现，LLMs利用这一向量与残差流结合来进行类似Word2Vec的向量运算，从而完成事实回忆型ICL任务。此外，近期的工作通过实证研究强调了问答数据在提升事实回忆能力方面的重要作用。尽管有了这些见解，但理论解释仍然缺失。 为了向前推进一步，我们提出了一种基于经验基础分层概念建模的理论框架。我们开发了一个优化理论，展示了非线性残差变压器如何通过对交叉熵损失进行梯度下降训练来通过向量运算执行事实回忆型ICL任务。我们证明了0-1损失收敛，并表明其具有强大的泛化能力，包括对概念重组和分布变化的鲁棒性。这些结果阐明了变压器相对于静态嵌入前体的优势。 实证模拟验证了我们的理论见解。
URL
https://arxiv.org/abs/2508.09820
