Abstract
We propose Bayesian optimal sequential prediction as a new principle for understanding in-context learning (ICL). Unlike interpretations framing Transformers as performing implicit gradient descent, we formalize ICL as meta-learning over latent sequence tasks. For tasks governed by Linear Gaussian State Space Models (LG-SSMs), we prove a meta-trained selective SSM asymptotically implements the Bayes-optimal predictor, converging to the posterior predictive mean. We further establish a statistical separation from gradient descent, constructing tasks with temporally correlated noise where the optimal Bayesian predictor strictly outperforms any empirical risk minimization (ERM) estimator. Since Transformers can be seen as performing implicit ERM, this demonstrates selective SSMs achieve lower asymptotic risk due to superior statistical efficiency. Experiments on synthetic LG-SSM tasks and a character-level Markov benchmark confirm selective SSMs converge faster to Bayes-optimal risk, show superior sample efficiency with longer contexts in structured-noise settings, and track latent states more robustly than linear Transformers. This reframes ICL from "implicit optimization" to "optimal inference," explaining the efficiency of selective SSMs and offering a principled basis for architecture design.
Abstract (translated)
我们提出了贝叶斯最优顺序预测作为理解上下文学习（ICL）的新原则。与将变压器解释为执行隐式梯度下降的框架不同，我们将ICL形式化为对潜在序列任务进行元学习。对于由线性高斯状态空间模型（LG-SSM）支配的任务，我们证明了经过元训练的选择性SSM在渐近意义上实现了贝叶斯最优预测器，并且收敛于后验预测均值。此外，我们在构建具有时间相关噪声的任务时建立了与梯度下降的统计分离，使得最佳贝叶斯预测器严格优于任何经验风险最小化（ERM）估计器。由于变压器可以视为执行隐式ERM，这表明选择性SSM通过更优的统计效率实现了更低渐近风险。在合成LG-SSM任务和字符级马尔可夫基准上的实验证实了选择性SSM更快地收敛于贝叶斯最优风险，在结构噪声环境中具有更好的样本效率，并且比线性变压器更稳健地跟踪潜在状态。这将ICL从“隐式优化”重新框架为“最佳推理”，解释了选择性SSM的高效性和架构设计的原则基础。
URL
https://arxiv.org/abs/2602.17744
