Abstract
Unsupervised Text Style Transfer (UTST) aims to build a system to transfer the stylistic properties of a given text without parallel text pairs. Compared with text transfer between style polarities, UTST for controllable intensity is more challenging due to the subtle differences in stylistic features across different intensity levels. Faced with the challenges posed by the lack of parallel data and the indistinguishability between adjacent intensity levels, we propose a SFT-then-PPO paradigm to fine-tune an LLM. We first fine-tune the LLM with synthesized parallel data. Then, we further train the LLM with PPO, where the rewards are elaborately designed for distinguishing the stylistic intensity in hierarchical levels. Both the global and local stylistic features are considered to formulate the reward functions. The experiments on two UTST benchmarks showcase that both rewards have their advantages and applying them to LLM fine-tuning can effectively improve the performance of an LLM backbone based on various evaluation metrics. Even for close levels of intensity, we can still observe the noticeable stylistic difference between the generated text.
Abstract (translated)
无监督文本风格迁移（UTST）的目标是构建一个系统，能够将给定文本的风格特征转移到其他文本中，而不需要平行文本对。与在不同风格极性之间进行文本转换相比，针对可控强度的UTST更具挑战性，因为不同强度级别的风格特征之间的差异非常细微。面对缺乏平行数据以及相邻强度级别难以区分的问题，我们提出了一种SFT-then-PPO（预训练后强化学习）范式来对大规模语言模型（LLM）进行微调。首先，使用合成的平行文本数据对LLM进行微调；然后，进一步利用PPO（Proximal Policy Optimization）方法训练该模型，其中奖励函数精心设计用于区分不同层次上的风格强度。在制定奖励函数时，既考虑了全局又考虑了局部的风格特征。 实验结果表明，在两个UTST基准测试上应用这两种类型的奖励均表现出各自的优点，并且将它们应用于LLM微调过程中能够有效提升基于多种评估指标下的模型性能。即使对于强度接近的情况下，仍然可以观察到生成文本之间的显著风格差异。
URL
https://arxiv.org/abs/2601.01060
