Abstract
Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes' theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities.
Abstract (translated)
大语言模型（LLMs）可以通过基于对话历史中提供的几个示例进行基于无模型参数更新的语境学习（ICL）来适应新的任务。尽管这种便利性，但ICL的性能很大程度上取决于提供的语境示例的质量，这使得语境示例选择方法成为一个关键的选择。本文提出了一种新颖的贝叶斯语境示例选择方法（ByCS）用于ICL。基于贝叶斯公式的推理概率条件，ByCS关注于基于测试输入的逆推理条件。假设准确的反向推理概率（概率）会导致准确的后验概率（后），根据逆推理结果选择语境示例。我们对语音、文本和图像例子进行了多样且广泛的跨任务和跨模态实验。实验结果展示了我们ByCS方法在不同模型、任务和模态上的有效性和鲁棒性。
URL
https://arxiv.org/abs/2404.14716
