Abstract
Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidentally, the NLP community has also recently found that self-generating relevant examples in the context can help large language models (LLMs) better solve a given problem than hand-crafted prompts. However, it is yet not clear whether relevance is the key factor eliciting such capability, i.e., can LLMs benefit more from self-generated relevant examples than irrelevant ones? In this work, we systematically explore whether LLMs can truly perform analogical reasoning on a diverse set of reasoning tasks. With extensive experiments and analysis, we show that self-generated random examples can surprisingly achieve comparable or even better performance, e.g., 4% performance boost on GSM8K with random biological examples. We find that the accuracy of self-generated examples is the key factor and subsequently design two improved methods with significantly reduced inference costs. Overall, we aim to advance a deeper understanding of LLM analogical reasoning and hope this work stimulates further research in the design of self-generated contexts.
Abstract (translated)
类比推理是人类独特的能力，通过将相关过去的经验策略应用于未知的挑战来解决问题。心理学的一个关键发现是，与无关的过去经验相比，回忆相关的经验可以帮助人类更好地处理新的任务。此外，自然语言处理（NLP）社区最近也发现，在特定背景下自生成相关的示例可以帮助大型语言模型（LLMs）比自手编写的提示更好地解决某个问题。然而，目前尚不清楚是否相关性是导致这种能力的关键因素，即LLM是否比无关的示例更有利？在这项工作中，我们系统地探讨LLMs是否可以在多样化的推理任务中进行类比推理。通过广泛的实验和分析，我们发现自生成随机示例可以意外地实现与自手编写的示例相当甚至更好的性能，例如在GSM8K上的性能提升达到4%。我们发现自生成示例的准确性是关键因素，因此我们设计了两项改进方法，降低了推理成本。总体而言，我们希望深入研究LLM的类比推理，并希望这项工作能激发进一步研究在自生成上下文的设计上。
URL
https://arxiv.org/abs/2404.12728
