Abstract
Fine-tuning pre-trained Large Language Models (LLMs) is essential to align them with human values and intentions. This process often utilizes methods like pairwise comparisons and KL divergence against a reference LLM, focusing on the evaluation of full answers generated by the models. However, the generation of these responses occurs in a token level, following a sequential, auto-regressive fashion. In this paper, we introduce Token-level Direct Preference Optimization (TDPO), a novel approach to align LLMs with human preferences by optimizing policy at the token level. Unlike previous methods, which face challenges in divergence efficiency, TDPO incorporates forward KL divergence constraints for each token, improving alignment and diversity. Utilizing the Bradley-Terry model for a token-based reward system, TDPO enhances the regulation of KL divergence, while preserving simplicity without the need for explicit reward modeling. Experimental results across various text tasks demonstrate TDPO's superior performance in balancing alignment with generation diversity. Notably, fine-tuning with TDPO strikes a better balance than DPO in the controlled sentiment generation and single-turn dialogue datasets, and significantly improves the quality of generated responses compared to both DPO and PPO-based RLHF methods. Our code is open-sourced at this https URL.
Abstract (translated)
微调预训练的大型语言模型（LLMs）与人类价值观和意图对齐至关重要。这一过程通常采用比较对等关系和与参考LLM的KL散度的方法，重点关注模型生成的完整答案的评估。然而，这些回答的生成是在标记级别进行的，遵循了序列、自回归的样式。在本文中，我们引入了Token-level Direct Preference Optimization（TDPO），一种通过优化模型在每个标记级别的策略来与人类偏好对齐的新颖方法。与之前的方法不同，TDPO通过每个标记点的正向KL散度约束来改善对齐和多样性。利用布拉德利-特里模型作为基于标记的奖励系统，TDPO增强了KL散度的规范，同时保留了简单性，无需显式奖励建模。在各种文本任务的各种实验结果中，TDPO在平衡对齐与生成多样性方面的表现优于DPO。值得注意的是，在受控情感生成和单轮对话数据集上，TDPO与DPO的微调效果略好于PPO，显著地提高了生成的响应的质量。我们的代码目前是开源的，在以下链接处。
URL
https://arxiv.org/abs/2404.11999
