Abstract
Large language models (LLMs) can acquire new knowledge through fine-tuning, but this process exhibits a puzzling duality: models can generalize remarkably from new facts, yet are also prone to hallucinating incorrect information. However, the reasons for this phenomenon remain poorly understood. In this work, we argue that both behaviors stem from a single mechanism known as out-of-context reasoning (OCR): the ability to deduce implications by associating concepts, even those without a causal link. Our experiments across five prominent LLMs confirm that OCR indeed drives both generalization and hallucination, depending on whether the associated concepts are causally related. To build a rigorous theoretical understanding of this phenomenon, we then formalize OCR as a synthetic factual recall task. We empirically show that a one-layer single-head attention-only transformer with factorized output and value matrices can learn to solve this task, while a model with combined weights cannot, highlighting the crucial role of matrix factorization. Our theoretical analysis shows that the OCR capability can be attributed to the implicit bias of gradient descent, which favors solutions that minimize the nuclear norm of the combined output-value matrix. This mathematical structure explains why the model learns to associate facts and implications with high sample efficiency, regardless of whether the correlation is causal or merely spurious. Ultimately, our work provides a theoretical foundation for understanding the OCR phenomenon, offering a new lens for analyzing and mitigating undesirable behaviors from knowledge injection.
Abstract (translated)
大型语言模型（LLMs）可以通过微调获得新知识，但这一过程表现出一种令人费解的二元性：这些模型可以从新的事实中进行显著地概括推理，但也容易产生不正确的信息。然而，导致这种现象的原因尚不清楚。在本项工作中，我们提出两种行为都源于单一机制，即跨背景推理（OCR）：即使没有因果关系，也能够通过关联概念推断出结论的能力。我们的实验涵盖了五个知名的大规模语言模型，并确认了 OCR 确实驱动着概括和幻觉的产生，这取决于相关联的概念是否具有因果联系。 为了建立这一现象的严谨理论理解，我们随后将 OCR 形式化为一个合成的事实回忆任务。通过实验，我们证明了一层单一头部注意力机制且输出和值矩阵因子分解的变压器能够学会解决该任务，而权重合并的模型则不能，这突显了矩阵分解的关键作用。 我们的理论分析表明，OCR 能力可以归因于梯度下降的隐含偏置，这种偏置倾向于选择使组合输出-值矩阵的核范数最小化的解决方案。这一数学结构解释了为什么模型能够以高样本效率学会将事实与结论关联起来，无论这些关联是因果关系还是仅仅偶然相关。 最终，我们的工作为理解 OCR 现象提供了理论基础，并提供了一种新的视角来分析和缓解知识注入带来的不良行为。
URL
https://arxiv.org/abs/2506.10887
