Abstract
Traditional reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed \textit{Self-Play Preference Optimization} (SPPO), approximates the Nash equilibrium through iterative policy updates and enjoys theoretical convergence guarantee. Our method can effectively increase the log-likelihood of the chosen response and decrease that of the rejected response, which cannot be trivially achieved by symmetric pairwise loss such as Direct Preference Optimization (DPO) and Identity Preference Optimization (IPO). In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models.
Abstract (translated)
传统强化学习从人类反馈（RLHF）方法依赖于参数模型（如Bradley-Terry模型）则无法捕捉到人类偏好中的可塑性和非理睬性。最近的研究表明，直接与偏好概率互动可能产生更准确的人类偏好的反映，从而实现更灵活和准确的语言模型对齐。在本文中，我们提出了一种基于自博弈的语言模型对齐方法，将问题视为一个恒等和两个玩家的无限博弈，旨在确定纳什均衡策略。我们的方法被称为\textit{自博弈偏好优化》（SPPO），通过迭代策略更新来近似纳什均衡，并具有理论收敛保证。我们的方法可以有效增加所选响应的似然，并减小拒绝响应的似然，这不能通过对称成对损失（如Direct Preference Optimization，DPO）和Identity Preference Optimization（IPO）等简单方式实现。在实验中，使用超Feedback集的60k个提示（没有响应）以及没有提示增强，仅利用预训练偏好模型PairRM，仅包含0.4B参数，SPPO可以从微调后的Mistral-7B-Instruct-v0.2模型中获得，该模型在AlpacaEval 2.0上的长度控制赢得率达到了28.53%。此外，SPPO在MT-Bench和Open LLM Leaderboard上的表现优于（迭代）DPO和IPO。值得注意的是，SPPO的高性能在没有额外外部监督（如响应、偏好等）的情况下，从GPT-4或其他更强的语言模型中实现。
URL
https://arxiv.org/abs/2405.00675
