Prediction of road users' behaviors in the context of autonomous driving has gained considerable attention by the scientific community in the last years. Most works focus on predicting behaviors based on kinematic information alone, a simplification of the reality since road users are humans, and as such they are highly influenced by their surrounding context. In addition, a large plethora of research works rely on powerful Deep Learning techniques, which exhibit high performance metrics in prediction tasks but may lack the ability to fully understand and exploit the contextual semantic information contained in the road scene, not to mention their inability to provide explainable predictions that can be understood by humans. In this work, we propose an explainable road users' behavior prediction system that integrates the reasoning abilities of Knowledge Graphs (KG) and the expressiveness capabilities of Large Language Models (LLM) by using Retrieval Augmented Generation (RAG) techniques. For that purpose, Knowledge Graph Embeddings (KGE) and Bayesian inference are combined to allow the deployment of a fully inductive reasoning system that enables the issuing of predictions that rely on legacy information contained in the graph as well as on current evidence gathered in real time by onboard sensors. Two use cases have been implemented following the proposed approach: 1) Prediction of pedestrians' crossing actions; 2) Prediction of lane change maneuvers. In both cases, the performance attained surpasses the current state of the art in terms of anticipation and F1-score, showing a promising avenue for future research in this field.
近年来,自动驾驶背景下预测道路使用者的行为已经引起了科学界的广泛关注。大多数工作都基于运动信息预测行为,简化现实,因为道路使用者是是人,所以他们对周围环境的影响很大。此外,大量研究作品依赖强大的深度学习技术,在预测任务中表现出高的性能指标,但可能无法完全理解并利用道路场景中的上下文语义信息,更不用说无法提供可解释的预测,让人类能够理解。在本文中,我们提出了一个可解释的道路使用者行为预测系统,通过使用检索增强生成(RAG)技术将知识图谱的推理能力和大型语言模型的表现力相结合。为此,知识图谱嵌入(KGE)和贝叶斯推理被结合使用,以便部署一个完全归纳推理系统,该系统能够基于图形中的旧信息以及车载传感器实时收集的证据发出预测。以下是根据所提出的方法实现的两个用例:1)预测行人过马路的行为;2)预测车道变更操作。在这两个用例中,取得的性能已经超越了当前的技术水平,显示了该领域未来研究的希望。
https://arxiv.org/abs/2405.00449
Distinguished from traditional knowledge graphs (KGs), temporal knowledge graphs (TKGs) must explore and reason over temporally evolving facts adequately. However, existing TKG approaches still face two main challenges, i.e., the limited capability to model arbitrary timestamps continuously and the lack of rich inference patterns under temporal constraints. In this paper, we propose an innovative TKGE method (PTBox) via polynomial decomposition-based temporal representation and box embedding-based entity representation to tackle the above-mentioned problems. Specifically, we decompose time information by polynomials and then enhance the model's capability to represent arbitrary timestamps flexibly by incorporating the learnable temporal basis tensor. In addition, we model every entity as a hyperrectangle box and define each relation as a transformation on the head and tail entity boxes. The entity boxes can capture complex geometric structures and learn robust representations, improving the model's inductive capability for rich inference patterns. Theoretically, our PTBox can encode arbitrary time information or even unseen timestamps while capturing rich inference patterns and higher-arity relations of the knowledge base. Extensive experiments on real-world datasets demonstrate the effectiveness of our method.
与传统知识图(KGs)相比,时间知识图(TKGs)必须充分探索和推理随时间变化的事实。然而,现有的TKG方法仍然面临着两个主要挑战,即无法连续建模任意时间戳以及缺乏在时间约束下丰富的推理模式。在本文中,我们提出了一个创新的时间知识图(TKGE)方法(PTBox),通过基于多项式的时本表示和基于箱嵌入的实体表示来解决上述问题。具体来说,我们通过多项式分解来分解时间信息,然后通过学习的时间本张量增强模型的能力来表示任意时间戳。此外,我们将每个实体建模为一个超矩形框,将每个关系建模为一个变换,该变换作用于头和尾实体框。实体框可以捕捉复杂的几何结构,并学习稳健的表示,提高模型对于复杂推理模式的归纳能力。从理论上将我们的PTBox可以编码任意时间信息或甚至未知的时刻,同时捕捉知识库中的丰富推理模式和高阶关系。在现实世界的数据集上进行大量实验证明了我们方法的有效性。
https://arxiv.org/abs/2405.00358
Temporal Knowledge Graph (TKG) reasoning often involves completing missing factual elements along the timeline. Although existing methods can learn good embeddings for each factual element in quadruples by integrating temporal information, they often fail to infer the evolution of temporal facts. This is mainly because of (1) insufficiently exploring the internal structure and semantic relationships within individual quadruples and (2) inadequately learning a unified representation of the contextual and temporal correlations among different quadruples. To overcome these limitations, we propose a novel Transformer-based reasoning model (dubbed ECEformer) for TKG to learn the Evolutionary Chain of Events (ECE). Specifically, we unfold the neighborhood subgraph of an entity node in chronological order, forming an evolutionary chain of events as the input for our model. Subsequently, we utilize a Transformer encoder to learn the embeddings of intra-quadruples for ECE. We then craft a mixed-context reasoning module based on the multi-layer perceptron (MLP) to learn the unified representations of inter-quadruples for ECE while accomplishing temporal knowledge reasoning. In addition, to enhance the timeliness of the events, we devise an additional time prediction task to complete effective temporal information within the learned unified representation. Extensive experiments on six benchmark datasets verify the state-of-the-art performance and the effectiveness of our method.
时空知识图(TKG)推理通常涉及在时间轴上完成缺失的事实元素。尽管现有的方法可以通过整合时间信息来学习每个事实元素的较好嵌入,但它们往往无法推断不同四元组之间的时空事实的演变。这主要是因为(1)对每个四元组内部结构和语义关系内部探索不足;(2)对不同四元组之间的上下文和时间关联的统一表示学习不足。为了克服这些限制,我们提出了一个新颖的Transformer基推理模型(被称为ECEformer),用于TKG学习演化链事件(ECE)。具体来说,我们按时间顺序展开实体节点周围的子图,将演化链事件作为输入,为我们的模型。随后,我们利用Transformer编码器学习 intra-quadruples 的嵌入。然后,我们根据多层感知器(MLP)构建混合上下文推理模块,学习不同四元组之间的统一表示,实现时间知识推理。此外,为了增强事件的及时性,我们还设计了一个额外的时间预测任务,以完成所学习到的统一表示中的有效时间信息。在六个基准数据集上的大量实验证实了我们的方法具有最先进的表现和效果。
https://arxiv.org/abs/2405.00352
Data protection and privacy is becoming increasingly crucial in the digital era. Numerous companies depend on third-party vendors and service providers to carry out critical functions within their operations, encompassing tasks such as data handling and storage. However, this reliance introduces potential vulnerabilities, as these vendors' security measures and practices may not always align with the standards expected by regulatory bodies. Businesses are required, often under the penalty of law, to ensure compliance with the evolving regulatory rules. Interpreting and implementing these regulations pose challenges due to their complexity. Regulatory documents are extensive, demanding significant effort for interpretation, while vendor-drafted privacy policies often lack the detail required for full legal compliance, leading to ambiguity. To ensure a concise interpretation of the regulatory requirements and compliance of organizational privacy policy with said regulations, we propose a Large Language Model (LLM) and Semantic Web based approach for privacy compliance. In this paper, we develop the novel Privacy Policy Compliance Verification Knowledge Graph, PrivComp-KG. It is designed to efficiently store and retrieve comprehensive information concerning privacy policies, regulatory frameworks, and domain-specific knowledge pertaining to the legal landscape of privacy. Using Retrieval Augmented Generation, we identify the relevant sections in a privacy policy with corresponding regulatory rules. This information about individual privacy policies is populated into the PrivComp-KG. Combining this with the domain context and rules, the PrivComp-KG can be queried to check for compliance with privacy policies by each vendor against relevant policy regulations. We demonstrate the relevance of the PrivComp-KG, by verifying compliance of privacy policy documents for various organizations.
数据保护和隐私在数字时代变得越来越重要。许多公司依赖第三方供应商和提供商执行运营中的关键功能,包括数据处理和存储。然而,这种依赖会引入潜在的安全漏洞,因为这些供应商的安全措施和做法可能并不总是符合监管机构的期望标准。根据法律要求,企业必须确保符合不断变化的监管规则。解释和实施这些法规由于其复杂性而面临挑战。监管文件涵盖广泛,需要大量精力进行解释,而供应商编写的隐私政策往往缺乏所需的详细信息,导致模糊不清。为了确保对监管要求和组织隐私政策的一致解释和合规,我们提出了一个基于大型语言模型(LLM)和语义网络的隐私合规方法。在本文中,我们开发了新颖的隐私政策合规验证知识图谱,PrivComp-KG。该知识图谱旨在有效地存储和检索关于隐私政策、监管框架和隐私法律领域的全面信息。通过检索增强生成,我们识别隐私政策中相应的法规条文。将个人隐私政策的信息充实到PrivComp-KG中。结合领域上下文和规则,PrivComp-KG可以用于检查每个供应商是否符合相关政策法规。我们通过验证各种组织的隐私政策文件对隐私政策的一致性进行验证,展示了PrivComp-KG的相關性。
https://arxiv.org/abs/2404.19744
External knowledge graphs (KGs) can be used to augment large language models (LLMs), while simultaneously providing an explainable knowledge base of facts that can be inspected by a human. This approach may be particularly valuable in domains where explainability is critical, like human trafficking data analysis. However, creating KGs can pose challenges. KGs parsed from documents may comprise explicit connections (those directly stated by a document) but miss implicit connections (those obvious to a human although not directly stated). To address these challenges, this preliminary research introduces the GAME-KG framework, standing for "Gaming for Augmenting Metadata and Enhancing Knowledge Graphs." GAME-KG is a federated approach to modifying explicit as well as implicit connections in KGs by using crowdsourced feedback collected through video games. GAME-KG is shown through two demonstrations: a Unity test scenario from Dark Shadows, a video game that collects feedback on KGs parsed from US Department of Justice (DOJ) Press Releases on human trafficking, and a following experiment where OpenAI's GPT-4 is prompted to answer questions based on a modified and unmodified KG. Initial results suggest that GAME-KG can be an effective framework for enhancing KGs, while simultaneously providing an explainable set of structured facts verified by humans.
外部知识图(KGs)可以用于增强大型语言模型(LLMs),同时为人类提供可检查的事实知识库。这种方法在需要解释性的领域(如打击人口贩卖数据分析)可能特别有价值。然而,创建KGs可能带来挑战。从文档中解析的KGs可能包括明确的连接(直接由文档中陈述的连接),但也会错过隐性的连接(对人类明显可见,尽管不是直接陈述的连接)。为解决这些挑战,这项初步研究引入了GAME-KG框架,它代表“通过游戏增强元数据和增强知识图”。GAME-KG是一种通过使用通过视频游戏收集的民办公众反馈修改KGs中明确和隐性连接的联邦方法。GAME-KG通过两个演示来说明:一个来自《黑暗影子》的Unity测试场景,该游戏收集了关于美国司法部(DOJ) press releases中的人口贩卖KGs的反馈,另一个是OpenAI的GPT-4被要求根据修改后的KG回答问题的实验。初步结果表明,GAME-KG可以成为增强KGs的有效框架,同时为人类提供经过验证的结构化事实知识库。
https://arxiv.org/abs/2404.19729
Trajectory prediction in autonomous driving relies on accurate representation of all relevant contexts of the driving scene including traffic participants, road topology, traffic signs as well as their semantic relations to each other. Despite increased attention to this issue, most approaches in trajectory prediction do not consider all of these factors sufficiently. This paper describes a method SemanticFormer to predict multimodal trajectories by reasoning over a semantic traffic scene graph using a hybrid approach. We extract high-level information in the form of semantic meta-paths from a knowledge graph which is then processed by a novel pipeline based on multiple attention mechanisms to predict accurate trajectories. The proposed architecture comprises a hierarchical heterogeneous graph encoder, which can capture spatio-temporal and relational information across agents and between agents and road elements, and a predictor that fuses the different encodings and decodes trajectories with probabilities. Finally, a refinement module evaluates permitted meta-paths of trajectories and speed profiles to obtain final predicted trajectories. Evaluation of the nuScenes benchmark demonstrates improved performance compared to the state-of-the-art methods.
轨迹预测在自动驾驶中依赖于准确地描述驾驶场景中所有相关上下文的轨迹,包括驾驶员、道路拓扑、交通标志以及它们之间的语义关系。尽管对这个问题给予了越来越多的关注,但大多数轨迹预测方法都没有考虑到所有这些因素。本文描述了一种名为 SemanticFormer 的方法,通过在语义交通场景图上进行推理来预测多模态轨迹。我们从一个知识图中提取高级信息,然后通过一种新颖的管道处理该知识图,预测准确的轨迹。所提出的架构包括分层异质图编码器,可以捕捉到车辆和道路元素以及它们之间的时空关系,以及一个预测器,将不同的编码器和解码轨迹的概率相结合。最后,一个优化模块评估了轨迹和速度剖面的允许元路径,以获得最终的预测轨迹。对 nuScenes 基准的评估显示,与最先进的方法相比,性能有所提高。
https://arxiv.org/abs/2404.19379
Knowledge graphs (KGs) are large datasets with specific structures representing large knowledge bases (KB) where each node represents a key entity and relations amongst them are typed edges. Natural language queries formed to extract information from a KB entail starting from specific nodes and reasoning over multiple edges of the corresponding KG to arrive at the correct set of answer nodes. Traditional approaches of question answering on KG are based on (a) semantic parsing (SP), where a logical form (e.g., S-expression, SPARQL query, etc.) is generated using node and edge embeddings and then reasoning over these representations or tuning language models to generate the final answer directly, or (b) information-retrieval based that works by extracting entities and relations sequentially. In this work, we evaluate the capability of (LLMs) to answer questions over KG that involve multiple hops. We show that depending upon the size and nature of the KG we need different approaches to extract and feed the relevant information to an LLM since every LLM comes with a fixed context window. We evaluate our approach on six KGs with and without the availability of example-specific sub-graphs and show that both the IR and SP-based methods can be adopted by LLMs resulting in an extremely competitive performance.
知识图(KGs)是具有特定结构的大型数据集,代表大量知识库(KB),其中每个节点表示一个关键实体,它们之间的关系是类型化的边。自然语言查询从特定的节点开始,通过推理与相应知识库中的多个边之间的关系,到达正确的答案节点。传统基于KG的问答方法是基于(a)语义解析(SP),其中使用节点和边嵌入生成逻辑形式(例如S-表达式、SPARQL查询等),然后在这些表示或调整语言模型的基础上进行推理,或(b)信息检索,该方法通过按顺序提取实体和关系来工作。 在这项工作中,我们评估了(LLMs)在回答涉及多个级的KG问题的能力。我们证明了,根据KG的大小和性质,我们需要不同的方法来提取和向LLM提供相关信息,因为每个LLM都具有固定的上下文窗口。我们在六个具有和没有例子特定子图的KG上评估我们的方法,并发现基于IR和SP的方法都可以被LLM采用,导致具有非常竞争力的性能。
https://arxiv.org/abs/2404.19234
Relational triple extraction is crucial work for the automatic construction of knowledge graphs. Existing methods only construct shallow representations from a token or token pair-level. However, previous works ignore local spatial dependencies of relational triples, resulting in a weakness of entity pair boundary detection. To tackle this problem, we propose a novel Region-based Table Filling method (RTF). We devise a novel region-based tagging scheme and bi-directional decoding strategy, which regard each relational triple as a region on the relation-specific table, and identifies triples by determining two endpoints of each region. We also introduce convolution to construct region-level table representations from a spatial perspective which makes triples easier to be captured. In addition, we share partial tagging scores among different relations to improve learning efficiency of relation classifier. Experimental results show that our method achieves state-of-the-art with better generalization capability on three variants of two widely used benchmark datasets.
关系三元提取对于知识图的自动构建至关重要。现有的方法仅从词或词对级别构建浅层次表示。然而,之前的 works忽略了关系三元组的局部空间依赖性,导致实体对边界检测的弱点。为了解决这个问题,我们提出了一个新颖的区域为基础的表填充方法(RTF)。我们设计了一个新颖的区域为基础的标签方案和双向解码策略,将每个关系三元组视为关系特定表中的一个区域,通过确定每个区域的两个端点来确定三元组。我们还引入卷积来从空间角度构建区域级别的表表示,使得三元组更容易被捕捉。此外,我们还在不同关系之间共享部分标签得分,以提高关系分类器的学习效率。实验结果表明,我们的方法在三个广泛使用基准数据集上实现了最先进的性能,且在扩展性方面表现出色。
https://arxiv.org/abs/2404.19154
Despite widespread applications of knowledge graphs (KGs) in various tasks such as question answering and intelligent conversational systems, existing KGs face two major challenges: information granularity and deficiency in timeliness. These hinder considerably the retrieval and analysis of in-context, fine-grained, and up-to-date knowledge from KGs, particularly in highly specialized themes (e.g., specialized scientific research) and rapidly evolving contexts (e.g., breaking news or disaster tracking). To tackle such challenges, we propose a theme-specific knowledge graph (i.e., ThemeKG), a KG constructed from a theme-specific corpus, and design an unsupervised framework for ThemeKG construction (named TKGCon). The framework takes raw theme-specific corpus and generates a high-quality KG that includes salient entities and relations under the theme. Specifically, we start with an entity ontology of the theme from Wikipedia, based on which we then generate candidate relations by Large Language Models (LLMs) to construct a relation ontology. To parse the documents from the theme corpus, we first map the extracted entity pairs to the ontology and retrieve the candidate relations. Finally, we incorporate the context and ontology to consolidate the relations for entity pairs. We observe that directly prompting GPT-4 for theme-specific KG leads to inaccurate entities (such as "two main types" as one entity in the query result) and unclear (such as "is", "has") or wrong relations (such as "have due to", "to start"). In contrast, by constructing the theme-specific KG step by step, our model outperforms GPT-4 and could consistently identify accurate entities and relations. Experimental results also show that our framework excels in evaluations compared with various KG construction baselines.
尽管知识图谱(KGs)在各种任务中的广泛应用,如问答和智能对话系统,现有KG面临两个主要挑战:信息粒度和时间不足。这些阻碍了从KGs中检索和分析上下文、细粒度和最新知识的能力,特别是在高度专业化的主题(例如,专业科学研究)和快速变化的环境(例如,新闻或灾害跟踪)中。为了应对这些挑战,我们提出了一个主题特定知识图(即 ThemeKG),一个基于主题特定语料库的知识图谱,并设计了用于 ThemeKG 构建的无监督框架(名为 TKGCon)。该框架从主题特定语料库中提取原始主题,然后通过大型语言模型(LLMs)生成候选关系,构建主题关系本体。为了解析主题语料库中的文档,我们首先将提取到的实体对映射到语料库,并检索候选关系。最后,我们将上下文和本体整合用于关系匹配。我们观察到,直接使用 GPT-4 生成主题特定 KG会导致不准确实体(例如查询结果中的“两个主要类型”作为一个实体),以及不清晰或错误的關係(例如“由於”或“开始于”)。相比之下,通过逐步构建主题特定 KG,我们的模型在比较各种 KG 建设基线方面表现出优异性能。实验结果还显示,我们的框架在各种 KG 建设基线上的评估中表现出色。
https://arxiv.org/abs/2404.19146
Knowledge graphs (KGs), which store an extensive number of relational facts (head, relation, tail), serve various applications. While many downstream tasks highly rely on the expressive modeling and predictive embedding of KGs, most of the current KG representation learning methods, where each entity is embedded as a vector in the Euclidean space and each relation is embedded as a transformation, follow an entity ranking protocol. On one hand, such an embedding design cannot capture many-to-many relations. On the other hand, in many retrieval cases, the users wish to get an exact set of answers without any ranking, especially when the results are expected to be precise, e.g., which genes cause an illness. Such scenarios are commonly referred to as "set retrieval". This work presents a pioneering study on the KG set retrieval problem. We show that the set retrieval highly depends on expressive modeling of many-to-many relations, and propose a new KG embedding model SpherE to address this problem. SpherE is based on rotational embedding methods, but each entity is embedded as a sphere instead of a vector. While inheriting the high interpretability of rotational-based models, our SpherE can more expressively model one-to-many, many-to-one, and many-to-many relations. Through extensive experiments, we show that our SpherE can well address the set retrieval problem while still having a good predictive ability to infer missing facts. The code is available at this https URL.
知识图(KGs)作为一种存储大量关系事实(头,关系,尾)的数据结构,具有各种应用价值。尽管许多下游任务高度依赖于KGs的表示建模和预测嵌入,但目前大多数KG表示学习方法,其中每个实体以欧氏空间中的向量表示,每个关系以变换表示,都遵循实体排序协议。一方面,这种嵌入设计无法捕捉许多对多关系。另一方面,在许多检索案例中,用户希望获得一个无排名的准确集合答案,尤其是在结果预计精确的情况下,例如哪些基因导致疾病。这种情况通常被称为“集检索”。 本文在KG集检索问题上进行了一项开创性的研究。我们证明了集检索高度依赖于多对多关系的表示建模,并提出了一个新的KG嵌入模型SpherE来解决这个问题。SpherE基于旋转嵌入方法,但每个实体都被嵌入为一个球体而不是向量。虽然继承了旋转模型的高可解释性,但我们的SpherE可以更富有表现力地建模一对一、一对多和多对多关系。通过大量实验,我们证明了我们的SpherE可以在解决集检索问题的同时,仍具有推断缺失事实的良好预测能力。代码可在此处访问:https://www.acm.org/dl/d/2222216
https://arxiv.org/abs/2404.19130
In customer service technical support, swiftly and accurately retrieving relevant past issues is critical for efficiently resolving customer inquiries. The conventional retrieval methods in retrieval-augmented generation (RAG) for large language models (LLMs) treat a large corpus of past issue tracking tickets as plain text, ignoring the crucial intra-issue structure and inter-issue relations, which limits performance. We introduce a novel customer service question-answering method that amalgamates RAG with a knowledge graph (KG). Our method constructs a KG from historical issues for use in retrieval, retaining the intra-issue structure and inter-issue relations. During the question-answering phase, our method parses consumer queries and retrieves related sub-graphs from the KG to generate answers. This integration of a KG not only improves retrieval accuracy by preserving customer service structure information but also enhances answering quality by mitigating the effects of text segmentation. Empirical assessments on our benchmark datasets, utilizing key retrieval (MRR, Recall@K, NDCG@K) and text generation (BLEU, ROUGE, METEOR) metrics, reveal that our method outperforms the baseline by 77.6% in MRR and by 0.32 in BLEU. Our method has been deployed within LinkedIn's customer service team for approximately six months and has reduced the median per-issue resolution time by 28.6%.
在客户服务技术支持中,迅速和准确地检索相关的历史问题对于高效解决客户咨询至关重要。对于大型语言模型(LLMs)的检索增强生成(RAG)中的传统检索方法,将大量历史问题跟踪 ticket 视为纯文本,忽略关键的内部问题结构和互操作关系,这会限制性能。我们引入了一种名为客户服务问题回答的新方法,将 RAG 与知识图(KG)相结合。我们的方法从历史问题中构建 KG,用于检索,保留内部问题结构和互操作关系。在问题回答阶段,我们的方法解析消费者查询并从 KG 中检索相关子图以生成答案。这种将 KG 与检索相结合的方法不仅通过保留客户服务结构信息提高了检索准确性,而且通过减轻文本分割的影响提高了回答质量。在我们基准数据集上进行的实证评估表明,使用关键检索(MRR,召回率@K,NDCG@K)和文本生成(BLEU,ROUGE,METEOR)指标评估,我们的方法在 MRR 和 BLEU 指标上均优于基线,分别高出基线 77.6% 和 0.32。我们的方法已经在 LinkedIn 的客户服务团队中部署了大约六个月,并将每件问题的中位数解决时间减少了 28.6%。
https://arxiv.org/abs/2404.17723
A backbone of knowledge graphs are their class membership relations, which assign entities to a given class. As part of the knowledge engineering process, we propose a new method for evaluating the quality of these relations by processing descriptions of a given entity and class using a zero-shot chain-of-thought classifier that uses a natural language intensional definition of a class. We evaluate the method using two publicly available knowledge graphs, Wikidata and CaLiGraph, and 7 large language models. Using the gpt-4-0125-preview large language model, the method's classification performance achieves a macro-averaged F1-score of 0.830 on data from Wikidata and 0.893 on data from CaLiGraph. Moreover, a manual analysis of the classification errors shows that 40.9% of errors were due to the knowledge graphs, with 16.0% due to missing relations and 24.9% due to incorrectly asserted relations. These results show how large language models can assist knowledge engineers in the process of knowledge graph refinement. The code and data are available on Github.
知识图谱的一个骨架是它们类成员关系,它们将实体分配给给定类。作为知识工程过程的一部分,我们提出了一种通过处理关于给定实体的描述和类使用零击链式思考类词表的类评估质量的新方法。我们使用两个公开可用的事务性知识图谱Wikidata和CaliGraph,以及7个大型语言模型来评估该方法。使用gpt-4-0125-preview大型语言模型,该方法的分类性能在Wikidata上的数据上实现了0.830的宏观平均F1分数,在CaliGraph上的数据上实现了0.893。此外,通过手动分析分类错误,我们发现40.9%的错误是由知识图谱引起的,其中16.0%是由于缺失关系,24.9%是由于错误声称的关系。这些结果表明,大型语言模型可以为知识工程师在知识图谱精炼过程中提供帮助。代码和数据可于Github上获取。
https://arxiv.org/abs/2404.17000
Our world is shaped by events of various complexity. This includes both small-scale local events like local farmer markets and large complex events like political and military conflicts. The latter are typically not observed directly but through the lenses of intermediaries like newspapers or social media. In other words, we do not witness the unfolding of such events directly but are confronted with narratives surrounding them. Such narratives capture different aspects of a complex event and may also differ with respect to the narrator. Thus, they provide a rich semantics concerning real-world events. In this paper, we show how narratives concerning complex events can be constructed and utilized. We provide a formal representation of narratives based on recursive nodes to represent multiple levels of detail and discuss how narratives can be bound to event-centric knowledge graphs. Additionally, we provide an algorithm based on incremental prompting techniques that mines such narratives from texts to account for different perspectives on complex events. Finally, we show the effectiveness and future research directions in a proof of concept.
我们的世界是由各种复杂事件塑造的。这包括小型的地方事件,如当地农民市场,以及大型复杂事件,如政治和军事冲突。后者的通常不是直接观察到的,而是通过中介机构,如报纸或社交媒体,透过他们的镜头观察到的。换句话说,我们不是直接见证了这些事件的展开,而是面对着围绕这些事件的故事。这些故事捕捉了复杂事件的不同方面,而且也可能与叙述者的观点有所不同。因此,它们提供了关于现实世界事件的丰富语义。在本文中,我们展示了如何构建和利用关于复杂事件的叙述。我们根据递归节点的形式给出了叙述的正式表示,并讨论了叙述如何与事件中心化的知识图谱相联系。此外,我们还基于增量提示技术提供了算法,用于从文本中挖掘这类叙述,以反映复杂事件的不同观点。最后,我们在概念证明中展示了这种方法的有效性和未来的研究方向。
https://arxiv.org/abs/2404.16405
Knowledge Graphs (KGs) are widely employed in artificial intelligence applications, such as question-answering and recommendation systems. However, KGs are frequently found to be incomplete. While much of the existing literature focuses on predicting missing nodes for given incomplete KG triples, there remains an opportunity to complete KGs by exploring relations between existing nodes, a task known as relation prediction. In this study, we propose a relations prediction model that harnesses both textual and structural information within KGs. Our approach integrates walks-based embeddings with language model embeddings to effectively represent nodes. We demonstrate that our model achieves competitive results in the relation prediction task when evaluated on a widely used dataset.
知识图(KGs)广泛应用于人工智能领域,如问答和推荐系统。然而,KGs经常被发现不完整。虽然现有文献主要关注预测给定不完整的KG三元组中的缺失节点,但在关系预测领域仍有机会通过探索现有节点之间的关系来完成KGs,实现名为关系预测的任务。在这项研究中,我们提出了一个利用KGs中文本和结构信息的关联预测模型。我们的方法结合了走行嵌入和语言模型嵌入,有效地表示节点。我们证明了,当在我们的广泛使用数据集上评估时,我们的模型在关系预测任务上实现了具有竞争力的结果。
https://arxiv.org/abs/2404.16206
The use of retrieval-augmented generation (RAG) to retrieve relevant information from an external knowledge source enables large language models (LLMs) to answer questions over private and/or previously unseen document collections. However, RAG fails on global questions directed at an entire text corpus, such as "What are the main themes in the dataset?", since this is inherently a query-focused summarization (QFS) task, rather than an explicit retrieval task. Prior QFS methods, meanwhile, fail to scale to the quantities of text indexed by typical RAG systems. To combine the strengths of these contrasting methods, we propose a Graph RAG approach to question answering over private text corpora that scales with both the generality of user questions and the quantity of source text to be indexed. Our approach uses an LLM to build a graph-based text index in two stages: first to derive an entity knowledge graph from the source documents, then to pregenerate community summaries for all groups of closely-related entities. Given a question, each community summary is used to generate a partial response, before all partial responses are again summarized in a final response to the user. For a class of global sensemaking questions over datasets in the 1 million token range, we show that Graph RAG leads to substantial improvements over a naïve RAG baseline for both the comprehensiveness and diversity of generated answers. An open-source, Python-based implementation of both global and local Graph RAG approaches is forthcoming at this https URL.
利用检索增强生成(RAG)从外部知识源中检索相关信息,使得大型语言模型(LLMs)能够回答从私有和/或之前未见过的文档集合中提出的问题。然而,RAG在指向整个文本语料库的全身问题时失败,例如“数据集的主要主题是什么?”,这是因为这是一个本质上的查询关注度总结(QFS)任务,而不是一个明确的检索任务。同时,之前的前QFS方法也无法扩展到典型RAG系统所索引的文本数量。为了结合这些相互矛盾的方法的优势,我们提出了一个基于图的RAG方法来在私有文本语料库上进行问题回答,该方法能够随着用户问题和要索引的源文本的数量而扩展。我们的方法使用LLM在两个阶段构建一个基于图的文本索引:首先从源文档中提取实体知识图,然后为所有密切相关的实体群组预生成社区摘要。对于一个问题,每个社区摘要用于生成部分回答,然后将所有部分回答再次汇总为对用户的最终回答。在100万词范围内的一类全局理解问题,我们证明了图RAG对于完整性和多样性生成的答案比 naive RAG 基线有了显著的改进。要在该https URL上找到一个开源的、基于全局和局部 Graph RAG方法的Python实现。
https://arxiv.org/abs/2404.16130
This study explores the use of Large Language Models (LLMs) for automatic evaluation of knowledge graph (KG) completion models. Historically, validating information in KGs has been a challenging task, requiring large-scale human annotation at prohibitive cost. With the emergence of general-purpose generative AI and LLMs, it is now plausible that human-in-the-loop validation could be replaced by a generative agent. We introduce a framework for consistency and validation when using generative models to validate knowledge graphs. Our framework is based upon recent open-source developments for structural and semantic validation of LLM outputs, and upon flexible approaches to fact checking and verification, supported by the capacity to reference external knowledge sources of any kind. The design is easy to adapt and extend, and can be used to verify any kind of graph-structured data through a combination of model-intrinsic knowledge, user-supplied context, and agents capable of external knowledge retrieval.
本研究探讨了使用大型语言模型(LLMs)自动评估知识图(KG)完成模型的应用。历史上,验证知识图中的有效信息是一个具有挑战性的任务,需要大规模的人类标注,代价高昂。随着通用生成式人工智能(GSA)和LLM的出现,现在可能用生成代理来代替人机交互验证。我们提出了一个使用生成模型验证知识图的一致性和验证框架。该框架基于LLM输出结构和语义验证的最近开源发展,以及支持外部知识来源访问的能力。该设计易于调整和扩展,可以通过模型固有知识、用户提供的上下文和支持外部知识检索的代理来验证任何类型的图状数据。
https://arxiv.org/abs/2404.15923
Knowledge Graph Completion (KGC) has garnered massive research interest recently, and most existing methods are designed following a transductive setting where all entities are observed during training. Despite the great progress on the transductive KGC, these methods struggle to conduct reasoning on emerging KGs involving unseen entities. Thus, inductive KGC, which aims to deduce missing links among unseen entities, has become a new trend. Many existing studies transform inductive KGC as a graph classification problem by extracting enclosing subgraphs surrounding each candidate triple. Unfortunately, they still face certain challenges, such as the expensive time consumption caused by the repeat extraction of enclosing subgraphs, and the deficiency of entity-independent feature learning. To address these issues, we propose a global-local anchor representation (GLAR) learning method for inductive KGC. Unlike previous methods that utilize enclosing subgraphs, we extract a shared opening subgraph for all candidates and perform reasoning on it, enabling the model to perform reasoning more efficiently. Moreover, we design some transferable global and local anchors to learn rich entity-independent features for emerging entities. Finally, a global-local graph reasoning model is applied on the opening subgraph to rank all candidates. Extensive experiments show that our GLAR outperforms most existing state-of-the-art methods.
知识图谱完成(KGC)最近吸引了大量研究兴趣,而且大多数现有方法都是在训练过程中采用转换设置的,其中所有实体都在观察过程中。尽管在转换式KGC方面取得了很大进展,但这些方法在处理涉及未见实体的 emergence KG 时仍然存在挑战。因此,归纳式KGC,旨在从未见实体中推断缺失链接,已成为一个新的趋势。许多现有研究将归纳式KGC转换为一个图分类问题,通过提取围绕每个候选三元组的内包容子图来完成。然而,他们仍然面临着某些挑战,例如由于重复提取内包容子图而产生的高时间消耗,以及实体独立特征学习不足的问题。为了解决这些问题,我们提出了一个全局-局部锚表示(GLAR)学习方法来解决归纳式KGC。与之前的方法不同,我们为所有候选者提取共享的内包容子图,并在其上进行推理,使模型能够更有效地进行推理。此外,我们还设计了一些可转移的全局和局部锚来学习新兴实体的丰富实体独立特征。最后,在打开子图上应用全局-局部图推理模型对所有候选者进行排名。大量实验证明,我们的GLAR超越了大多数现有最先进的方法。
https://arxiv.org/abs/2404.15807
This paper develops an innovative method that enables neural networks to generate and utilize knowledge graphs, which describe their concept-level knowledge and optimize network parameters through alignment with human-provided knowledge. This research addresses a gap where traditionally, network-generated knowledge has been limited to applications in downstream symbolic analysis or enhancing network transparency. By integrating a novel autoencoder design with the Vector Symbolic Architecture (VSA), we have introduced auxiliary tasks that support end-to-end training. Our approach eschews traditional dependencies on ontologies or word embedding models, mining concepts from neural networks and directly aligning them with human knowledge. Experiments show that our method consistently captures network-generated concepts that align closely with human knowledge and can even uncover new, useful concepts not previously identified by humans. This plug-and-play strategy not only enhances the interpretability of neural networks but also facilitates the integration of symbolic logical reasoning within these systems.
本文提出了一种创新的方法,使神经网络能够生成并利用知识图谱,该知识图谱描述了其概念层面的知识,并通过与人类提供的知识对齐来优化网络参数。这项研究解决了传统情况下,网络生成的知识仅限于下游符号分析或增强网络透明性的应用之间。通过将新颖的自动编码器设计集成到向量符号架构(VSA)中,我们引入了支持端到端训练的辅助任务。我们的方法跳出了传统对元数据或词嵌入模型的依赖,从神经网络中挖掘概念,并直接与人类知识对齐。实验证明,我们的方法能够捕获与人类知识密切相关的大规模网络生成概念,甚至能够发现之前未被人类发现的新的有用的概念。这种可插拔策略不仅增强了神经网络的可解释性,而且还促进了这些系统中符号逻辑推理的集成。
https://arxiv.org/abs/2404.16884
Graphs play an important role in representing complex relationships in various domains like social networks, knowledge graphs, and molecular discovery. With the advent of deep learning, Graph Neural Networks (GNNs) have emerged as a cornerstone in Graph Machine Learning (Graph ML), facilitating the representation and processing of graph structures. Recently, LLMs have demonstrated unprecedented capabilities in language tasks and are widely adopted in a variety of applications such as computer vision and recommender systems. This remarkable success has also attracted interest in applying LLMs to the graph domain. Increasing efforts have been made to explore the potential of LLMs in advancing Graph ML's generalization, transferability, and few-shot learning ability. Meanwhile, graphs, especially knowledge graphs, are rich in reliable factual knowledge, which can be utilized to enhance the reasoning capabilities of LLMs and potentially alleviate their limitations such as hallucinations and the lack of explainability. Given the rapid progress of this research direction, a systematic review summarizing the latest advancements for Graph ML in the era of LLMs is necessary to provide an in-depth understanding to researchers and practitioners. Therefore, in this survey, we first review the recent developments in Graph ML. We then explore how LLMs can be utilized to enhance the quality of graph features, alleviate the reliance on labeled data, and address challenges such as graph heterogeneity and out-of-distribution (OOD) generalization. Afterward, we delve into how graphs can enhance LLMs, highlighting their abilities to enhance LLM pre-training and inference. Furthermore, we investigate various applications and discuss the potential future directions in this promising field.
图在表示复杂关系方面在社交网络、知识图谱和分子发现等领域中发挥着重要作用。随着深度学习的出现,图神经网络(GNNs)成为图机器学习(Graph ML)的一个支柱,推动了图结构的表示和处理。近年来,LLM在语言任务上的表现已经达到了史无前例的水平,并在各种应用领域(如计算机视觉和推荐系统)得到了广泛应用。这一显著的成功也引起了将LLM应用于图形领域的兴趣。越来越多的努力致力于探索LLM在推动图机器学习的一般化、可迁移性和少样本学习能力方面的潜力。同时,特别是知识图谱,图形在可靠的事实知识方面非常丰富,可以利用来增强LLM的推理能力,并可能减轻其局限性,如幻觉和缺乏可解释性。鉴于这一研究领域的快速进步,对于LLM时代图机器学习的系统综述总结最新的进展是必要的,以提供研究人员和实践者对这一领域的深入理解。因此,在本次调查中,我们首先回顾了图机器学习领域的最新发展。然后,我们探讨了LLM如何用于提高图形特征的质量、减轻对标注数据的依赖以及解决诸如图形异质性和离散(OOD)泛化等问题。接着,我们深入研究了图形如何增强LLM,强调了它们在提高LLM预训练和推理能力方面的能力。最后,我们调查了各种应用,并讨论了这一充满前景的领域未来的潜在方向。
https://arxiv.org/abs/2404.14928
A graph is a fundamental data model to represent various entities and their complex relationships in society and nature, such as social networks, transportation networks, financial networks, and biomedical systems. Recently, large language models (LLMs) have showcased a strong generalization ability to handle various NLP and multi-mode tasks to answer users' arbitrary questions and specific-domain content generation. Compared with graph learning models, LLMs enjoy superior advantages in addressing the challenges of generalizing graph tasks by eliminating the need for training graph learning models and reducing the cost of manual annotation. In this survey, we conduct a comprehensive investigation of existing LLM studies on graph data, which summarizes the relevant graph analytics tasks solved by advanced LLM models and points out the existing remaining challenges and future directions. Specifically, we study the key problems of LLM-based generative graph analytics (LLM-GGA) with three categories: LLM-based graph query processing (LLM-GQP), LLM-based graph inference and learning (LLM-GIL), and graph-LLM-based applications. LLM-GQP focuses on an integration of graph analytics techniques and LLM prompts, including graph understanding and knowledge graph (KG) based augmented retrieval, while LLM-GIL focuses on learning and reasoning over graphs, including graph learning, graph-formed reasoning and graph representation. We summarize the useful prompts incorporated into LLM to handle different graph downstream tasks. Moreover, we give a summary of LLM model evaluation, benchmark datasets/tasks, and a deep pro and cons analysis of LLM models. We also explore open problems and future directions in this exciting interdisciplinary research area of LLMs and graph analytics.
图形是一种基本的数据模型,用于表示社会和自然中各种实体及其复杂的关系,如社交网络、交通网络、金融网络和生物医学系统。近年来,大型语言模型(LLMs)在处理各种自然语言处理(NLP)和多模态任务方面表现出强大的泛化能力,从而回答用户的任意问题和特定领域内容生成。与图形学习模型相比,LLMs在解决图形任务的挑战方面具有优越的优势,通过消除训练图形学习模型的需求并降低手动注释的成本。在本次调查中,我们对LLM关于图形数据的现有研究进行全面调查,概述了高级LLM模型解决的相关图形分析任务,并指出了现有的剩余挑战和未来发展方向。具体来说,我们研究了基于LLM的生成图数据分析(LLM-GGA)的三个主要问题:LLM-基于图查询处理(LLM-GQP)、LLM-基于图推理和学习(LLM-GIL)和基于图形-LLM的应用。LLM-GQP关注将图形数据分析技术和LLM提示进行集成,包括基于图理解和知识图(KG)的增强检索,而LLM-GIL关注在图形上进行学习和推理,包括图形学习、图形形成推理和图形表示。我们总结了LLM中纳入不同图形下游任务的有用提示。此外,我们还对LLM模型评估、基准数据集/任务以及LLM模型的优缺点进行了总结。此外,我们在LLM和图数据分析这一激动人心的跨学科研究领域中进行了探索。
https://arxiv.org/abs/2404.14809