Abstract
Knowledge graphs play a pivotal role in various applications, such as question-answering and fact-checking. Abstract Meaning Representation (AMR) represents text as knowledge graphs. Evaluating the quality of these graphs involves matching them structurally to each other and semantically to the source text. Existing AMR metrics are inefficient and struggle to capture semantic similarity. We also lack a systematic evaluation benchmark for assessing structural similarity between AMR graphs. To overcome these limitations, we introduce a novel AMR similarity metric, rematch, alongside a new evaluation for structural similarity called RARE. Among state-of-the-art metrics, rematch ranks second in structural similarity; and first in semantic similarity by 1--5 percentage points on the STS-B and SICK-R benchmarks. Rematch is also five times faster than the next most efficient metric.
Abstract (translated)
知识图在各种应用中扮演着关键角色，如问答和事实核查。抽象意义表示（AMR）将文本表示为知识图。评估这些图的质量涉及将它们结构上对齐，语义上与源文本对齐。现有的AMR指标效率低下，且很难捕捉语义相似性。此外，我们还没有一个系统性的评估基准来评估AMR图之间的结构相似性。为了克服这些限制，我们引入了一个新的AMR相似度指标——rematch，并引入了一个新的评估结构相似性的基准——RARE。在最先进的指标中，rematch在结构相似性上排名第二；而在STS-B和SICK-R基准上，它在语义相似性上比第二高效的指标高1-5个百分点。rematch也比下一个最有效的指标快五倍。
URL
https://arxiv.org/abs/2404.02126
