Abstract
LLMs make it easy to rewrite text in any style, be it more polite, persuasive, or more positive. We present a large-scale study of evaluation metrics for style and attribute transfer with a focus on content preservation; meaning content not attributed to the style shift is preserved. The de facto evaluation approach uses lexical or semantic similarity metrics often between source sentences and rewrites. While these metrics are not designed to distinguish between style or content differences, empirical meta-evaluation shows a reasonable correlation to human judgment. In fact, recent works find that LLMs prompted as evaluators are only comparable to semantic similarity metrics, even though intuitively, the LLM approach should better fit the task. To investigate this discrepancy, we benchmark 8 metrics for evaluating content preservation on existing datasets and additionally construct a new test set that better aligns with the meta-evaluation aim. Indeed, we then find that the empirical conclusion aligns with the intuition: content preservation metrics for style/attribute transfer must be conditional on the style shift. To support this, we propose a new efficient zero-shot evaluation method using the likelihood of the next token. We hope our meta-evaluation can foster more research on evaluating content preservation metrics, and also to ensure fair evaluation of methods for conducting style transfer.
Abstract (translated)
大型语言模型（LLMs）使得以任何风格重写文本变得容易，无论是更加礼貌、更具说服力还是更积极。我们提出了一项大规模的研究，评估用于风格和属性转换的评价指标，重点关注内容保留；也就是说，在不涉及风格变化的情况下，内容被保持不变。目前的事实标准评估方法使用词汇或语义相似性度量，通常是在源句子与重写后的句子之间进行比较。尽管这些度量不是为区分风格或内容差异而设计的，但经验元评价显示它们与人类判断有相当合理的相关性。事实上，最近的研究发现，作为评估者的LLMs在性能上仅相当于语义相似性度量，虽然从直觉上看，LLM的方法应该更适合这一任务。为了调查这种不一致的原因，我们在现有的数据集上对8种用于评估内容保留的指标进行了基准测试，并额外构建了一个新的测试集以更好地与元评价目标相符合。确实，在此之后我们发现经验结论与直觉相符：风格/属性转换的内容保留度量必须基于风格转变条件化。为此，我们提出了一种使用下一个标记可能性的新零样本评估方法，该方法更为高效。我们希望我们的元评价可以促进更多关于内容保留指标的评估研究，并确保对进行样式转换的方法进行公平评估。
URL
https://arxiv.org/abs/2502.15022
