Abstract
Recognizing semantic differences across documents, especially in different languages, is crucial for text generation evaluation and multilingual content alignment. However, as a standalone task it has received little attention. We address this by introducing SwissGov-RSD, the first naturalistic, document-level, cross-lingual dataset for semantic difference recognition. It encompasses a total of 224 multi-parallel documents in English-German, English-French, and English-Italian with token-level difference annotations by human annotators. We evaluate a variety of open-source and closed source large language models as well as encoder models across different fine-tuning settings on this new benchmark. Our results show that current automatic approaches perform poorly compared to their performance on monolingual, sentence-level, and synthetic benchmarks, revealing a considerable gap for both LLMs and encoder models. We make our code and datasets publicly available.
Abstract (translated)
识别不同文档之间的语义差异，尤其是在不同的语言中，对于文本生成评估和多语言内容对齐至关重要。然而，作为一个独立的任务，它并没有得到太多的关注。我们通过引入SwissGov-RSD来解决这个问题，这是第一个旨在跨语言语义差异识别的自然化、文档级数据集，包括总共224份在英语-德语、英语-法语和英语-意大利语之间的多平行文档，并由人工标注员提供了词级别差异注释。我们在这个新的基准上评估了各种开源和闭源的大规模语言模型以及不同微调设置下的编码器模型。我们的结果表明，当前的自动方法在与单语种、句子级和合成基准相比时表现较差，显示出大规模语言模型（LLMs）和编码器模型在这方面的显著差距。我们公开发布了代码和数据集。
URL
https://arxiv.org/abs/2512.07538
