Abstract
We present a theory-first framework that interprets inference-time adaptation in large language models (LLMs) as online Bayesian state estimation. Rather than modeling rapid adaptation as implicit optimization or meta-learning, we formulate task- and context-specific learning as the sequential inference of a low-dimensional latent adaptation state governed by a linearized state-space model. Under Gaussian assumptions, adaptation follows a Kalman recursion with closed-form updates for both the posterior mean and covariance. This perspective elevates epistemic uncertainty to an explicit dynamical variable. We show that inference-time learning is driven by covariance collapse, i.e., rapid contraction of posterior uncertainty induced by informative tokens, which typically precedes convergence of the posterior mean. Using observability conditions on token-level Jacobians, we establish stability of the Bayesian filter, prove exponential covariance contraction rates, and derive mean-square error bounds. Gradient descent, natural-gradient methods, and meta-learning updates arise as singular, noise-free limits of the filtering dynamics, positioning optimization-based adaptation as a degenerate approximation of Bayesian inference. The resulting theory provides a unified probabilistic account of in-context learning, parameter-efficient adaptation, and test-time learning without parameter updates. It yields explicit guarantees on stability and sample efficiency, offers a principled interpretation of prompt informativeness via information accumulation, and clarifies the role of uncertainty dynamics absent from existing accounts. Minimal illustrative experiments corroborate the qualitative predictions of the theory.
Abstract (translated)
我们提出了一种基于理论的框架，将大型语言模型（LLMs）在推理时的适应性解释为在线贝叶斯状态估计。不同于将快速适应视为隐式优化或元学习建模的方式，我们将特定任务和上下文的学习视为由线性化状态空间模型支配的低维潜在适应状态的顺序推断过程。在高斯假设下，适应遵循具有后验均值和协方差闭合形式更新的卡尔曼递归。这种视角将认识论不确定性提升为一个显式的动态变量。 我们表明，在推理时间学习是由协方差坍缩驱动的，即由信息令牌引起的后验不确定性的快速收缩，通常发生在后验均值收敛之前。通过在标记级别雅可比矩阵上的可观测性条件，我们建立了贝叶斯滤波器的稳定性，证明了指数级的协方差收缩率，并推导出了均方误差界限。 梯度下降、自然梯度方法以及元学习更新作为过滤动力学中无噪声极限的情况出现，将基于优化的方法适应视为贝叶斯推理的一种退化近似。由此产生的理论为上下文中的学习提供了一个统一的概率解释，涵盖了参数效率高且无需参数更新的测试时间学习，并提供了关于稳定性和样本效率的具体保证。 该理论通过信息积累的方式对提示信息性进行了原则性的诠释，并澄清了在现有论述中缺乏描述的不确定性动态的角色。最小的说明性实验验证了该理论的定性预测。
URL
https://arxiv.org/abs/2601.06100
