Abstract
In recent years, pre-trained large language models have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. However, existing literature has highlighted the sensitivity of this capability to the selection of few-shot demonstrations. The underlying mechanisms by which this capability arises from regular language model pretraining objectives remain poorly understood. In this study, we aim to examine the in-context learning phenomenon through a Bayesian lens, viewing large language models as topic models that implicitly infer task-related information from demonstrations. On this premise, we propose an algorithm for selecting optimal demonstrations from a set of annotated data and demonstrate a significant 12.5% improvement relative to the random selection baseline, averaged over eight GPT2 and GPT3 models on eight different real-world text classification datasets. Our empirical findings support our hypothesis that large language models implicitly infer a latent concept variable.
Abstract (translated)
近年来,训练的大型语言模型在实现Inference-time few-shot学习功能方面表现出显著的效率,这一功能被称为上下文学习。然而,现有的文献已经强调了这种能力对少量示例选择敏感性的重要性。从概率论的角度看,这种能力从Regular Language Model pretraining objectives产生的机制仍然不太理解。在本研究中,我们希望通过贝叶斯视角看待上下文学习现象,将大型语言模型视为主题模型,从示例中 implicit infer 任务相关的信息。基于这一假设,我们提出了一种算法,从一组标注数据中选择最优示例,并表明与随机选择基准相比,平均而言在八个不同的真实文本分类数据集上提高了12.5%。我们的实验结果支持我们的假设,即大型语言模型 implicit infer 一个隐含义变量。
URL
https://arxiv.org/abs/2301.11916
