Abstract
Natural language explanations have become a proxy for evaluating explainable and multi-step Natural Language Inference (NLI) models. However, assessing the validity of explanations for NLI is challenging as it typically involves the crowd-sourcing of apposite datasets, a process that is time-consuming and prone to logical errors. To address existing limitations, this paper investigates the verification and refinement of natural language explanations through the integration of Large Language Models (LLMs) and Theorem Provers (TPs). Specifically, we present a neuro-symbolic framework, named Explanation-Refiner, that augments a TP with LLMs to generate and formalise explanatory sentences and suggest potential inference strategies for NLI. In turn, the TP is employed to provide formal guarantees on the logical validity of the explanations and to generate feedback for subsequent improvements. We demonstrate how Explanation-Refiner can be jointly used to evaluate explanatory reasoning, autoformalisation, and error correction mechanisms of state-of-the-art LLMs as well as to automatically enhance the quality of human-annotated explanations of variable complexity in different domains.
Abstract (translated)
自然语言解释已成为评估可解释性和多步骤自然语言推理（NLI）模型的指标。然而，评估解释的有效性具有挑战性，因为它通常涉及apprise数据的众包，这个过程费时且容易出错。为了应对现有局限，本文研究了通过整合大型语言模型（LLMs）和定理证明器（TPs）来验证和优化自然语言解释的方法。具体来说，我们提出了一个名为Explanation-Refiner的神经符号框架，该框架通过在TP中添加LLMs来生成和形式化解释性句子，并建议可能的NLI推理策略。TP则用于提供关于解释逻辑有效性的正式保证，并生成后续改进的反馈。我们证明了Explanation-Refiner可以与现有LLM的推理、自动形式化和错误纠正机制共同用于评估可解释推理、自动形式化和不同领域的变体解释的质量。
URL
https://arxiv.org/abs/2405.01379
