Abstract
Direct preference optimization (DPO) is an effective technique to train language models to generate preferred over dispreferred responses. However, this binary "winner-takes-all" approach is suboptimal for vision-language models whose response quality is highly dependent on visual content. In particular, a response may still be faithful to the visual inputs even if it is less preferable than an alternative. The standard Bradley-Terry DPO formulation lacks this nuance, upweighting winning responses without sufficient regard for whether the "losing" response still maintains high visual fidelity. In this work, we investigate ranking optimization as an alternative that more precisely situates responses' faithfulness to visual inputs. We focus on video-text alignment using detailed video captions, proposing a method to generate challenging, totally ordered caption chains at scale through repeated caption degradation. Our results show ranking optimization outperforms binary DPO for long-form content generation and assessment, and importantly, we find that these approaches require finetuning of the vision encoder to be effective, challenging the view of DPO as purely a language-reweighting process.
Abstract (translated)
直接偏好优化（DPO）是一种有效的技术，用于训练语言模型以生成优于次优响应的内容。然而，这种二元“赢家通吃”的方法对于视觉-语言模型而言并非最优，因为后者的响应质量高度依赖于视觉内容。具体而言，即使某个响应不如替代方案理想，它仍可能对视觉输入保持忠实。标准的Bradley-Terry DPO公式缺乏这种细微差别，它在过度加权获胜响应的同时，未充分关注“失败”响应是否仍维持较高的视觉保真度。在本研究中，我们探索将排序优化作为替代方案，以更精确地定位响应与视觉输入之间的忠实度关系。我们聚焦于使用详细视频字幕的视频-文本对齐任务，提出了一种通过重复字幕降解来大规模生成具有挑战性的完全有序字幕链的方法。结果表明，排序优化在长内容生成与评估中优于二元DPO。更重要的是，我们发现这些方法需要微调视觉编码器才能发挥效果，这挑战了将DPO视为纯粹语言重加权过程的观点。
URL
https://arxiv.org/abs/2603.25145
