Abstract
To address the issue of insufficient knowledge and the tendency to generate hallucination in Large Language Models (LLMs), numerous studies have endeavored to integrate LLMs with Knowledge Graphs (KGs). However, all these methods are evaluated on conventional Knowledge Graph Question Answering (KGQA) with complete KGs, where the factual triples involved in each question are entirely covered by the given KG. In this situation, LLM mainly acts as an agent to find answer entities by exploring the KG, rather than effectively integrating internal and external knowledge sources. However, in real-world scenarios, KGs are often incomplete to cover all the knowledge required to answer questions. To simulate real-world scenarios and evaluate the ability of LLMs to integrate internal and external knowledge, in this paper, we propose leveraging LLMs for QA under Incomplete Knowledge Graph (IKGQA), where the given KG doesn't include all the factual triples involved in each question. To handle IKGQA, we propose a training-free method called Generate-on-Graph (GoG) that can generate new factual triples while exploring on KGs. Specifically, we propose a selecting-generating-answering framework, which not only treat the LLM as an agent to explore on KGs, but also treat it as a KG to generate new facts based on the explored subgraph and its inherent knowledge. Experimental results on two datasets demonstrate that our GoG can solve IKGQA to a certain extent, while almost all previous methods cannot perform well on IKGQA.
Abstract (translated)
为解决大型语言模型（LLMs）中知识不足和产生虚构现象的问题，许多研究努力将LLMs与知识图谱（KGs）集成。然而，这些方法都在传统的知识图谱问题回答（KGQA）上进行评估，其中每个问题涉及的事实三元组完全由给定的KG覆盖。在这种情况下，LLM主要作为探索KG的代理来查找答案实体，而不是有效地将内部和外部知识来源集成。然而，在现实世界的场景中，知识图谱通常不完整，无法覆盖所有回答问题的知识。为了模拟现实世界的场景，评估LLM将内部和外部知识整合的能力，本文提出了一种利用LLM进行基于不完整知识图谱（IKGQA）的QA的方法，其中给定的KG不包含每个问题涉及的所有事实三元组。为了处理IKGQA，我们提出了一种无需训练的方法，称为生成-在图（GoG）方法，可以在探索KGs的同时生成新的事实三元组。具体来说，我们提出了一种选择-生成-回答框架，不仅将LLM视为在KGs中探索的代理，还将它视为一个生成新事实的KG，基于探索的子图及其固有知识。在两个数据集上的实验结果表明，我们的GoG可以在一定程度上解决IKGQA，而几乎所有以前的方法在IKGQA上表现不佳。
URL
https://arxiv.org/abs/2404.14741
