Abstract
Leveraging external knowledge to enhance the reasoning ability is crucial for commonsense question answering. However, the existing knowledge bases heavily rely on manual annotation which unavoidably causes deficiency in coverage of world-wide commonsense knowledge. Accordingly, the knowledge bases fail to be flexible enough to support the reasoning over diverse questions. Recently, large-scale language models (LLMs) have dramatically improved the intelligence in capturing and leveraging knowledge, which opens up a new way to address the issue of eliciting knowledge from language models. We propose a Unified Facts Obtaining (UFO) approach. UFO turns LLMs into knowledge sources and produces relevant facts (knowledge statements) for the given question. We first develop a unified prompt consisting of demonstrations that cover different aspects of commonsense and different question styles. On this basis, we instruct the LLMs to generate question-related supporting facts for various commonsense questions via prompting. After facts generation, we apply a dense retrieval-based fact selection strategy to choose the best-matched fact. This kind of facts will be fed into the answer inference model along with the question. Notably, due to the design of unified prompts, UFO can support reasoning in various commonsense aspects (including general commonsense, scientific commonsense, and social commonsense). Extensive experiments on CommonsenseQA 2.0, OpenBookQA, QASC, and Social IQA benchmarks show that UFO significantly improves the performance of the inference model and outperforms manually constructed knowledge sources.
Abstract (translated)
利用外部知识增强推理能力对于常识问题回答至关重要。然而，现有的知识库严重依赖手动标注，这不可避免地会导致全球常识知识覆盖面的短缺。因此，知识库无法变得足够灵活，支持在各种问题上进行推理。最近，大型语言模型(LLMs)已经显著改进了捕获和利用知识的智商，这开辟了一种新方式，以从语言模型中获取知识。我们提出了一种统一的获取事实(UFO)方法。 UFO将LLMs转化为知识库，为给定问题生成相关的事实(知识陈述)。我们首先开发一个统一的提示，包括演示，涵盖了常识的不同方面和不同问题风格。基于这个提示，我们指令LLMs通过提示生成与问题相关的支持事实，以各种常识问题为例。在事实生成后，我们应用密集检索based的事实选择策略，选择最佳的匹配事实。这种类型的事实将随着问题一起输入答案推理模型。值得注意的是，由于统一提示的设计， UFO可以支持各种常识方面(包括一般常识、科学常识和社会常识)的推理。在常识问题QA 2.0、OpenBookQA、QASC和社交IQA基准实验中，广泛测试表明， UFO显著提高了推理模型的性能，并击败了手动构建的知识库。
URL
https://arxiv.org/abs/2305.16048
