Abstract
Curriculum learning plays a crucial role in enhancing the training efficiency of large language models (LLMs) on reasoning tasks. However, existing methods often fail to adequately account for variations in prompt difficulty or rely on simplistic filtering mechanisms to select prompt datasets within a narrow criterion range, resulting in significant computational waste. In this work, we approach the problem from the perspective of reinforcement learning gradient optimization, offering a systematic and theoretical investigation into how to improve the training efficiency of LLMs. We identify two key factors influencing training efficiency: the selection of training prompts and the allocation of rollout quantities across different prompts. Our theoretical analysis reveals that the sampling distribution of prompts dictates the convergence rate of gradient descent, while the allocation of the rollout quantity influences the consistency and stability of overall gradient updates. Based on these insights, we propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead. Experiments demonstrate that our CurES outperforms Group Relative Policy Optimization (GRPO) by \textbf{+3.30} points and \textbf{+4.82} points with 1.5B and 7B models, respectively. Additionally, CurES exhibits faster convergence compared to baselines, including GRPO.
Abstract (translated)
课程学习在提高大规模语言模型（LLM）在推理任务上的训练效率方面起着关键作用。然而，现有的方法常常无法充分考虑到提示难度的差异，或者依赖于简单的过滤机制来选择特定标准范围内的提示数据集，这导致了显著的计算浪费。在这项工作中，我们从强化学习梯度优化的角度出发，对如何提高LLM的训练效率进行了系统的理论研究。我们确定了影响训练效率的两个关键因素：训练提示的选择和不同提示下rollout数量的分配。我们的理论分析揭示了提示采样分布决定了梯度下降的收敛速度，而rollout数量的分配则影响整体梯度更新的一致性和稳定性。 基于这些见解，我们提出了CurES这一高效的训练方法，它通过加速收敛并采用贝叶斯后验估计来最小化计算开销。实验结果表明，在1.5B和7B模型上，我们的CurES方法分别比Group Relative Policy Optimization (GRPO) 方法高出3.30分和4.82分。此外，与包括GRPO在内的基线相比，CurES展示了更快的收敛速度。
URL
https://arxiv.org/abs/2510.01037
