Abstract
The automatic extraction of information is important for populating large web knowledge bases such as Wikidata. The temporal version of that task, temporal knowledge graph extraction (TKGE), involves extracting temporally grounded facts from text, represented as semantic quadruples (subject, relation, object, timestamp). Many recent systems take advantage of large language models (LLMs), which are becoming a new cornerstone of the web due to their performance on many tasks across the natural language processing (NLP) field. Despite the importance of TKGE, existing datasets for training and evaluation remain scarce, and contamination of evaluation data is an unaddressed issue, potentially inflating LLMs' perceived performance due to overlaps between training and evaluation sets. To mitigate these challenges, we propose a novel synthetic evaluation dataset constructed from predicted future, previously unseen temporal facts, thereby eliminating contamination and enabling robust and unbiased benchmarking. Our dataset creation involves a two-step approach: (1) Temporal Knowledge Graph Forecasting (TKGF) generates plausible future quadruples, which are subsequently filtered to adhere to the original knowledge base schema; (2) LLMs perform quadruple-to-text generation, creating semantically aligned textual descriptions. We benchmark Extract, Define and Canonicalize (EDC), a state-of-the-art LLM-based extraction framework, demonstrating that LLM performance decreases when evaluated on our dataset compared to a dataset of known facts. We publicly release our dataset consisting of 4.2K future quadruples and corresponding textual descriptions, along with the generation methodology, enabling continuous creation of unlimited future temporal datasets to serve as long-term, contamination-free benchmarks for TKGE.
Abstract (translated)
信息的自动提取对于填充如Wikidata这样的大型网络知识库至关重要。该任务的时间版本，即时间知识图谱抽取（Temporal Knowledge Graph Extraction，TKGE），涉及从文本中抽取带有时间戳记的事实，并将其表示为语义四元组（主体、关系、客体、时间戳）。许多最近的系统利用了大规模语言模型（Large Language Models, LLMs），这些模型由于在自然语言处理（NLP）领域的众多任务中的出色表现，正在成为网络的新基石。尽管TKGE的重要性不言而喻，但现有的用于训练和评估的数据集仍然匮乏，并且评价数据受到污染的问题未得到解决，这可能会因为训练集与验证集之间的重叠而导致LLMs的性能被夸大。为了解决这些问题，我们提出了一种新的合成评估数据集，该数据集由预测未来、以前从未见过的时间事实构成，从而消除了污染并能够进行稳健且无偏见的基准测试。 我们的数据集创建采用两步方法：(1) 时间知识图谱预测（Temporal Knowledge Graph Forecasting, TKGF）生成合理的未来四元组，并随后对其进行过滤以符合原有的知识库模式；(2) LLMs执行四元组到文本的生成，创建语义一致的文本描述。我们对Extract、Define and Canonicalize (EDC)，一种基于LLM的最先进的提取框架进行了基准测试，发现与已知事实的数据集相比，在我们的数据集中评估时LLMs的表现有所下降。 我们将公开发布包含4200个未来四元组及其对应文本描述的数据集，并附上生成方法，这将有助于持续创建无限量的未来的、无污染的时间数据集，作为长期基准测试供TKGE使用。
URL
https://arxiv.org/abs/2601.13658
