Abstract
Recent knowledge enhanced pre-trained language models have shown remarkable performance on downstream tasks by incorporating structured knowledge from external sources into language models. However, they usually suffer from a heterogeneous information alignment problem and a noisy knowledge injection problem. For complex reasoning, the contexts contain rich knowledge that typically exists in complex and sparse forms. In order to model structured knowledge in the context and avoid these two problems, we propose to unify structure reasoning and language model pre-training. It identifies four types of elementary knowledge structures from contexts to construct structured queries, and utilizes the box embedding method to conduct explicit structure reasoning along queries during language modeling. To fuse textual and structured semantics, we utilize contextual language representations of knowledge structures to initialize their box embeddings for structure reasoning. We conduct experiments on complex language reasoning and knowledge graph (KG) reasoning tasks. The results show that our model can effectively enhance the performance of complex reasoning of both language and KG modalities.
Abstract (translated)
最近的知识增强预训练语言模型通过将来自外部 sources 的结构化知识嵌入语言模型,在后续任务中表现出卓越的性能。然而,它们通常面临不一致信息对齐和噪声知识注入的问题。对于复杂推理,上下文中包含丰富的知识,通常以复杂和稀疏的形式存在。为了在上下文中建模结构化知识并避免这些问题,我们建议将结构推理和语言模型预训练统一起来。它从上下文中识别四种基本知识结构,构建结构化查询,并在语言建模期间使用Box嵌入方法进行明确的结构推理。为了融合文本和结构化语义,我们使用上下文知识结构的语言表示初始化它们的Box嵌入用于结构推理。我们进行了复杂的语言推理和知识图(KG)推理任务的实验。结果表明,我们的模型能够有效增强语言和KGmodality 的复杂推理性能。
URL
https://arxiv.org/abs/2301.08913
