Abstract
Large reasoning models (LRMs) generate intermediate reasoning traces before producing final answers, yielding strong gains on multi-step and mathematical tasks. Yet aligning LRMs with human preferences, a crucial prerequisite for model deployment, remains underexplored. The statistically correct objective for preference alignment requires marginalizing over reasoning traces, but this computation is intractable in practice. A common workaround optimizes a single sampled trajectory, which introduces substantial gradient variance from stochastic trace sampling. To address this challenge, we frame preference optimization for LRMs through the lens of the bias--variance trade-off and propose Bias--Variance Optimized Preference Optimization (BVPO), a simple, drop-in method that mixes two gradient estimators: a high-variance trace-based estimator and a low-variance empty-trace estimator obtained by disabling reasoning trace generation. Our theory shows that BVPO strictly reduces trace-induced variance for any nontrivial mixture, provides a closed-form choice of the mixing weight that minimizes mean-squared error relative to the true marginal gradient, and under standard smoothness and step-size conditions, tightens classical convergence bounds for stochastic gradient descent. Empirically, BVPO improves alignment over the best baseline by up to 7.8 points on AlpacaEval~2 and 6.8 points on Arena-Hard. Despite being trained only on general conversational data, BVPO also boosts reasoning performance for base models by up to 4.0 points on the average of six math reasoning benchmarks. These results identify variance from trace sampling as a key bottleneck and demonstrate that directly optimizing the bias--variance trade-off yields more stable training and stronger overall performance.
Abstract (translated)
大型推理模型（LRMs）在生成最终答案之前会先生成中间的推理痕迹，这在多步骤和数学任务中表现出色。然而，将这些模型与人类偏好对齐仍是一个未被充分研究的问题，这是部署模型的一个关键前提条件。统计上正确的对齐目标需要对推理痕迹进行边缘化处理，但在实践中这种计算是不可行的。常见的替代方案是优化单一抽样轨迹，但这引入了从随机痕迹采样中产生的大量梯度方差。 为了解决这个问题，我们通过偏差-方差权衡的角度来审视LRMs中的偏好优化，并提出了偏差-方差优化的偏好优化（BVPO），这是一种简单的方法，可以在不改变原有模型结构的情况下使用。该方法结合了两种梯度估计器：高方差痕迹基估计器和低方差空痕估计器（通过禁用推理痕迹生成获得）。我们的理论表明，对于任何非平凡混合物，BVPO可以严格降低由痕迹引起的方差，并提供了一个闭式的权重选择形式，以最小化相对于真实边际梯度的均方误差。在标准平滑性和步长条件下，它还可以紧缩经典随机梯度下降的收敛界限。 实证研究显示，在AlpacaEval~2和Arena-Hard任务中，BVPO比最佳基线提高了最多7.8分和6.8分。即使只用一般的对话数据进行训练，BVPO也能够提升基础模型在六个数学推理基准测试平均分数上的表现，最多提升了4.0分。 这些结果表明，从痕迹采样中产生的方差是主要的瓶颈问题，并证明了直接优化偏差-方差权衡可以带来更稳定的训练和更强的整体性能。
URL
https://arxiv.org/abs/2510.05095
