Abstract
Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task--data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and human evaluation.
Abstract (translated)
强化学习来自人类反馈（RLHF）和像直接偏好优化（DPO）这样的导数技术是用于将通用基础模型适应特定任务的任务对齐算法。我们证明了将任务对齐应用于自然语言翻译（NMT）解决了NMT中存在的任务-数据不匹配问题，从而在多语言模型的所有语言上改善了任务完成情况。为此，我们引入了直接质量优化（DQO），一种利用预训练翻译质量估计模型作为人类偏好的代理，并通过自动指标和人类评估来验证改进。
URL
https://arxiv.org/abs/2409.17673
