Abstract
Alignment and adaptation in large language models heavily rely on reinforcement learning from human feedback (RLHF); yet, theoretical understanding of its generalisability remains premature, especially when the learned reward could shift, and the KL control is estimated and clipped. To address this issue, we develop generalisation theory for RLHF that explicitly accounts for (1) \emph{reward shift}: reward models are trained on preference data from earlier or mixed behaviour policies while RLHF optimises the current policy on its own rollouts; and (2) \emph{clipped KL regularisation}: the KL regulariser is estimated from sampled log-probability ratios and then clipped for stabilisation, resulting in an error to RLHF. We present generalisation bounds for RLHF, suggesting that the generalisation error stems from a sampling error from prompts and rollouts, a reward shift error, and a KL clipping error. We also discuss special cases of (1) initialising RLHF parameters with a uniform prior over a finite space, and (2) training RLHF by stochastic gradient descent, as an Ornstein-Uhlenbeck process. The theory yields practical implications in (1) optimal KL clipping threshold, and (2) budget allocation in prompts, rollouts, and preference data.
Abstract (translated)
大型语言模型的对齐和适应很大程度上依赖于来自人类反馈的强化学习（RLHF），然而，关于其泛化能力的理论理解仍处于初级阶段，尤其是在学到的奖励可能发生变化且KL控制被估计并裁剪的情况下。为了解决这一问题，我们开发了针对RLHF的泛化理论，该理论明确考虑了以下两点： 1. **奖励漂移（Reward Shift）**：奖励模型是在早期或混合行为策略的数据上训练的，而RLHF则是基于当前策略在自身生成的rollout数据进行优化。 2. **裁剪KL正则化（Clipped KL Regularisation）**：KL正则项是从抽样对数概率比估计得到，并随后被裁剪以稳定算法，这会导致RLHF中的误差。 我们提出了针对RLHF的泛化边界，表明其泛化错误源自于从提示和rollout抽样的样本误差、奖励漂移引起的误差以及KL裁剪导致的误差。此外，还讨论了两个特殊情形： 1. RLHF参数初始化为有限空间上的均匀先验。 2. 通过随机梯度下降（SGD）训练RLHF，视其为奥恩斯坦-乌伦贝克过程。 该理论在实践中具有重要的意义，包括确定最佳的KL裁剪阈值以及如何分配提示、rollout和偏好数据的成本预算。
URL
https://arxiv.org/abs/2602.21765
