Abstract
Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences. However, effectively aligning LLMs with diverse human preferences remains a significant challenge, particularly when they are conflict. To address this issue, we frame human value alignment as a multi-objective optimization problem, aiming to maximize a set of potentially conflicting objectives. We introduce Gradient-Adaptive Policy Optimization (GAPO), a novel fine-tuning paradigm that employs multiple-gradient descent to align LLMs with diverse preference distributions. GAPO adaptively rescales the gradients for each objective to determine an update direction that optimally balances the trade-offs between objectives. Additionally, we introduce P-GAPO, which incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user's specific needs. Our theoretical analysis demonstrates that GAPO converges towards a Pareto optimal solution for multiple objectives. Empirical results on Mistral-7B show that GAPO outperforms current state-of-the-art methods, achieving superior performance in both helpfulness and harmlessness.
Abstract (translated)
从人类反馈中的强化学习（RLHF）已成为将大型语言模型（LLM）与人类偏好对齐的强大技术。然而，有效地使LLM与多样化的甚至是冲突的人类偏好相一致仍是一个重大挑战。为了解决这个问题，我们将人类价值观的对齐问题构架成一个多目标优化问题，旨在最大化一组潜在冲突的目标。我们引入了一种新型微调范式——梯度自适应策略优化（GAPO），它利用多梯度下降来使LLM与多种偏好分布相一致。GAPO能够根据每个目标自适应地调整梯度的尺度，以确定一个最优平衡各目标间权衡的更新方向。此外，我们还引入了P-GAPO，该方法结合不同目标上的用户偏好，并实现帕累托解，更好地满足用户的特定需求。理论分析表明，GAPO可以朝着多目标问题的帕累托最优解收敛。在Mistral-7B模型上的实证结果显示，GAPO超越了当前最先进的方法，在有用性和无害性方面表现出色。
URL
https://arxiv.org/abs/2507.01915
