Abstract
In-context learning (ICL) has transformed the use of large language models (LLMs) for NLP tasks, enabling few-shot learning by conditioning on labeled examples without finetuning. Despite its effectiveness, ICL is prone to errors, especially for challenging examples. With the goal of improving the performance of ICL, we propose corrective in-context learning (CICL), an approach that incorporates a model's incorrect predictions alongside ground truth corrections into the prompt, aiming to enhance classification accuracy through self-correction. However, contrary to our hypothesis, extensive experiments on text classification tasks demonstrate that CICL consistently underperforms standard ICL, with performance degrading as the proportion of corrections in the prompt increases. Our findings indicate that CICL introduces confusion by disrupting the model's task understanding, rather than refining its predictions. Additionally, we observe that presenting harder examples in standard ICL does not improve performance, suggesting that example difficulty alone may not be a reliable criterion for effective selection. By presenting these negative results, we provide important insights into the limitations of self-corrective mechanisms in LLMs and offer directions for future research.
Abstract (translated)
基于上下文的学习（ICL）已经改变了大型语言模型（LLMs）在自然语言处理（NLP）任务中的使用方式，通过利用标注示例进行少量样本学习而不需微调。尽管ICL非常有效，但它容易出错，尤其是在面对难题时。为了改进ICL的性能，我们提出了校正上下文学习（CICL），这是一种将模型错误预测和真实纠正结果整合到提示中的方法，旨在通过自我修正来提高分类精度。然而，与我们的假设相反，在文本分类任务上的大量实验表明，CICL的表现始终逊于标准ICL，并且随着提示中纠错比例的增加，性能还会下降。我们的研究发现指出，CICL引入了混淆，破坏了模型的任务理解能力，而不是改进其预测结果。此外，我们还观察到在标准ICL中呈现更难的例子并不能提高表现，这表明难度本身可能不是有效选择示例的一个可靠标准。通过展示这些负面结果，我们为LLMs自我修正机制的局限性提供了重要见解，并为进一步研究指明了方向。
URL
https://arxiv.org/abs/2503.16022
