Abstract
Large Language Models (LLMs) have made significant strides in information acquisition. However, their overreliance on potentially flawed parametric knowledge leads to hallucinations and inaccuracies, particularly when handling long-tail, domain-specific queries. Retrieval Augmented Generation (RAG) addresses this limitation by incorporating external, non-parametric knowledge. Nevertheless, the retrieved long-context documents often contain noisy, irrelevant information alongside vital knowledge, negatively diluting LLMs' attention. Inspired by the supportive role of essential concepts in individuals' reading comprehension, we propose a novel concept-based RAG framework with the Abstract Meaning Representation (AMR)-based concept distillation algorithm. The proposed algorithm compresses the cluttered raw retrieved documents into a compact set of crucial concepts distilled from the informative nodes of AMR by referring to reliable linguistic features. The concepts explicitly constrain LLMs to focus solely on vital information in the inference process. We conduct extensive experiments on open-domain question-answering datasets to empirically evaluate the proposed method's effectiveness. The results indicate that the concept-based RAG framework outperforms other baseline methods, particularly as the number of supporting documents increases, while also exhibiting robustness across various backbone LLMs. This emphasizes the distilled concepts are informative for augmenting the RAG process by filtering out interference information. To the best of our knowledge, this is the first work introducing AMR to enhance the RAG, presenting a potential solution to augment inference performance with semantic-based context compression.
Abstract (translated)
大语言模型（LLMs）在信息获取方面取得了显著的进展。然而，他们对可能存在缺陷的参数知识的过度依赖导致了一种虚幻和不准确的情况，特别是在处理长尾和领域特定问题时。检索增强生成（RAG）通过引入外部、非参数性知识来解决这个局限。然而，检索到的长文本文档中通常包含噪音，无关信息，这会削弱LLM的注意力。为了模仿人在阅读理解中的关键概念对个人阅读理解的积极作用，我们提出了一个基于概念的RAG框架，使用基于抽象意义表示（AMR）的观念蒸馏算法。该算法通过参考AMR的信息节点将混杂的原始检索文档压缩成关键概念蒸馏的紧凑集合。这些概念明确约束LLM仅在推理过程中关注关键信息。我们在开放域问题回答数据集上进行广泛的实验，以实证评估所提出的方法的有效性。实验结果表明，基于概念的RAG框架在其他基线方法中表现出优异的表现，特别是在支持文档数量增加时，同时还表现出对各种基线LLM的稳健性。这强调了蒸馏出来的概念对增强RAG过程具有信息滤波作用。据我们所知，这是第一个将AMR引入增强RAG的工作，提出了用语义基于上下文压缩来提高推理性能的潜在解决方案。
URL
https://arxiv.org/abs/2405.03085
