Abstract
Traditional knowledge graph embedding (KGE) methods typically require preserving the entire knowledge graph (KG) with significant training costs when new knowledge emerges. To address this issue, the continual knowledge graph embedding (CKGE) task has been proposed to train the KGE model by learning emerging knowledge efficiently while simultaneously preserving decent old knowledge. However, the explicit graph structure in KGs, which is critical for the above goal, has been heavily ignored by existing CKGE methods. On the one hand, existing methods usually learn new triples in a random order, destroying the inner structure of new KGs. On the other hand, old triples are preserved with equal priority, failing to alleviate catastrophic forgetting effectively. In this paper, we propose a competitive method for CKGE based on incremental distillation (IncDE), which considers the full use of the explicit graph structure in KGs. First, to optimize the learning order, we introduce a hierarchical strategy, ranking new triples for layer-by-layer learning. By employing the inter- and intra-hierarchical orders together, new triples are grouped into layers based on the graph structure features. Secondly, to preserve the old knowledge effectively, we devise a novel incremental distillation mechanism, which facilitates the seamless transfer of entity representations from the previous layer to the next one, promoting old knowledge preservation. Finally, we adopt a two-stage training paradigm to avoid the over-corruption of old knowledge influenced by under-trained new knowledge. Experimental results demonstrate the superiority of IncDE over state-of-the-art baselines. Notably, the incremental distillation mechanism contributes to improvements of 0.2%-6.5% in the mean reciprocal rank (MRR) score.
Abstract (translated)
传统知识图嵌入（KGE）方法通常需要在新的知识出现时付出巨大的训练成本来保留整个知识图（KG）。为解决这个问题，连续知识图嵌入（CKGE）任务被提出，通过在同时学习和保留旧知识的同时，以高效的方式训练KGE模型。然而，现有的CKGE方法对KGs的显式图结构的重大忽视。一方面，现有方法通常在学习过程中以随机顺序学习新的三元组，破坏了新KG的内部结构。另一方面，旧三元组以等同的优先级被保留，未能有效减轻灾难性遗忘。在本文中，我们提出了一个基于增量的 distillation（IncDE）的竞争性的CKGE 方法，该方法考虑了 KGs 的显式图结构的全面利用。首先，为了优化学习顺序，我们引入了一个层次策略，对每个层级的新的三元组进行排序。通过同时使用上下文和内部层次结构，将新的三元组分组到基于图结构特征的层中。其次，为了有效地保留旧知识，我们设计了一种新颖的增量式蒸馏机制，促进了从前一层到下一层的实体表示的平稳转移，促进了旧知识的保留。最后，我们采用两级训练范式来避免受到欠训练的新知识中旧知识的过度污染。实验结果表明，与最先进的基线相比，IncDE 的优越性得到了充分证明。值得注意的是，增量式蒸馏机制对平均互反排名（MRR）得分提高了0.2%-6.5%。
URL
https://arxiv.org/abs/2405.04453
