Abstract
Recently developed large language models (LLMs) have been shown to perform remarkably well on a wide range of language understanding tasks. But, can they really "reason" over the natural language? This question has been receiving significant research attention and many reasoning skills such as commonsense, numerical, and qualitative have been studied. However, the crucial skill pertaining to 'logical reasoning' has remained underexplored. Existing work investigating this reasoning ability of LLMs has focused only on a couple of inference rules (such as modus ponens and modus tollens) of propositional and first-order logic. Addressing the above limitation, we comprehensively evaluate the logical reasoning ability of LLMs on 25 different reasoning patterns spanning over propositional, first-order, and non-monotonic logics. To enable systematic evaluation, we introduce LogicBench, a natural language question-answering dataset focusing on the use of a single inference rule. We conduct detailed analysis with a range of LLMs such as GPT-4, ChatGPT, Gemini, Llama-2, and Mistral using chain-of-thought prompting. Experimental results show that existing LLMs do not fare well on LogicBench; especially, they struggle with instances involving complex reasoning and negations. Furthermore, they sometimes overlook contextual information necessary for reasoning to arrive at the correct conclusion. We believe that our work and findings facilitate future research for evaluating and enhancing the logical reasoning ability of LLMs. Data and code are available at this https URL.
Abstract (translated)
近年来发展的大型语言模型（LLMs）在各种语言理解任务上的表现确实非常出色。但是，它们能否真正“理性”地处理自然语言呢？这个问题受到了大量的研究关注，并且已经研究了很多推理技能，如常识、数量推理和定性推理。然而，关于“逻辑推理”这一关键技能，迄今为止仍缺乏深入的研究。现有工作在研究LLMs的推理能力时，仅关注了命题和第一级逻辑的几个推理规则（如模态推理和推论规则）。为解决这一局限，我们全面评估了LLMs在25个不同的推理模式上的逻辑推理能力，这些模式跨越了命题、第一级逻辑和非规范逻辑。为了进行系统性的评估，我们引入了LogicBench，这是一个关注使用单一推理规则的自然语言问题回答数据集。我们使用包括GPT-4、ChatGPT、Gemini、Llama-2和Mistral在内的各种LLM，使用连锁思考提示进行详细分析。实验结果表明，现有LLM在LogicBench的表现不佳；尤其是，它们在涉及复杂推理和否定实例时表现不佳。此外，它们有时会忽视推理所需的语言上下文信息。我们认为，我们的工作及其成果有助于未来研究为评估和提高LLMs的逻辑推理能力提供方向。数据和代码可在此https://www.academia.edu/39411041/Logic_Reasoning_for_LLMs_LogicBench_and_Beyond_with_Chain_of_Thought_Prompting_Towards_a_Systematic_Evaluation
URL
https://arxiv.org/abs/2404.15522
