Abstract
Text Style Transfer (TST) is the task of transforming a text to reflect a particular style while preserving its original content. Evaluating TST outputs is a multidimensional challenge, requiring the assessment of style transfer accuracy, content preservation, and naturalness. Using human evaluation is ideal but costly, same as in other natural language processing (NLP) tasks, however, automatic metrics for TST have not received as much attention as metrics for, e.g., machine translation or summarization. In this paper, we examine both set of existing and novel metrics from broader NLP tasks for TST evaluation, focusing on two popular subtasks-sentiment transfer and detoxification-in a multilingual context comprising English, Hindi, and Bengali. By conducting meta-evaluation through correlation with human judgments, we demonstrate the effectiveness of these metrics when used individually and in ensembles. Additionally, we investigate the potential of Large Language Models (LLMs) as tools for TST evaluation. Our findings highlight that certain advanced NLP metrics and experimental-hybrid-techniques, provide better insights than existing TST metrics for delivering more accurate, consistent, and reproducible TST evaluations.
Abstract (translated)
文本风格转换（TST）的任务是将一段文本转化为具有特定风格的版本，同时保留其原始内容。评估TST输出是一个多维度挑战，需要对风格转化准确性、内容保存和自然度进行评测。虽然人工评价是最理想的但成本高昂，与其他自然语言处理（NLP）任务类似，自动化的TST评估指标并未像机器翻译或摘要生成等领域那样受到足够重视。在本文中，我们考察了现有以及新提出的广泛应用于其他NLP任务的评估标准，并集中于两个流行的子任务：情感转换和去毒化，在包含英语、印地语和孟加拉语的多语言环境中进行研究。通过与人工判断的相关性测试作为元评价方法，我们展示了这些指标在单独使用及组合使用时的有效性。此外，我们也探讨了大型语言模型（LLMs）作为TST评估工具的潜力。我们的研究表明，某些先进的NLP评估标准和实验混合技术比现有的TST评估标准能提供更准确、一致且可重复的结果。
URL
https://arxiv.org/abs/2502.04718
