Abstract
We investigate the potential of ChatGPT as a multidimensional evaluator for the task of \emph{Text Style Transfer}, alongside, and in comparison to, existing automatic metrics as well as human judgements. We focus on a zero-shot setting, i.e. prompting ChatGPT with specific task instructions, and test its performance on three commonly-used dimensions of text style transfer evaluation: style strength, content preservation, and fluency. We perform a comprehensive correlation analysis for two transfer directions (and overall) at different levels. Compared to existing automatic metrics, ChatGPT achieves competitive correlations with human judgments. These preliminary results are expected to provide a first glimpse into the role of large language models in the multidimensional evaluation of stylized text generation.
Abstract (translated)
我们研究了 ChatGPT 作为文本风格转移任务多因素评估器的潜力，并与之相比，同时考虑了现有的自动 metrics 和人类判断。我们重点关注了一个零次响应设置，即通过特定的任务指令prompt ChatGPT，并测试它在三个常见的文本风格转移评估维度上的表现：风格强度、内容保留和流畅度。我们在两个转移方向(以及整体)上进行了全面的相关性分析。与现有的自动 metrics 相比，ChatGPT 实现了竞争性相关性。这些初步结果预计将提供一个初步瞥见大型语言模型在格式化文本生成多因素评估中的作用。
URL
https://arxiv.org/abs/2304.13462
