Abstract
Recent advancements in Large Language Models (LLMs) have demonstrated that Process Reward Models (PRMs) play a crucial role in enhancing model performance. However, training PRMs typically requires step-level labels, either manually annotated or automatically generated, which can be costly and difficult to obtain at scale. To address this challenge, we introduce FreePRM, a weakly supervised framework for training PRMs without access to ground-truth step-level labels. FreePRM first generates pseudo step-level labels based on the correctness of final outcome, and then employs Buffer Probability to eliminate impact of noise inherent in pseudo labeling. Experimental results show that FreePRM achieves an average F1 score of 53.0% on ProcessBench, outperforming fully supervised PRM trained on Math-Shepherd by +24.1%. Compared to other open-source PRMs, FreePRM outperforms upon RLHFlow-PRM-Mistral-8B (28.4%) by +24.6%, EurusPRM (31.3%) by +21.7%, and Skywork-PRM-7B (42.1%) by +10.9%. This work introduces a new paradigm in PRM training, significantly reducing reliance on costly step-level annotations while maintaining strong performance.
Abstract (translated)
最近在大型语言模型（LLMs）方面取得的进展表明，过程奖励模型（PRMs）在提高模型性能方面发挥着关键作用。然而，训练PRMs通常需要逐步骤的标签数据，这些标签要么是人工标注的，要么是由自动化工具生成的，这既昂贵又难以大规模获取。为了解决这一挑战，我们引入了FreePRM，这是一种无需访问真实步级标签就能训练PRM的弱监督框架。 FreePRM首先基于最终结果的正确性生成伪步骤级标签，然后利用缓存概率来消除伪标记中固有的噪声影响。实验结果显示，在ProcessBench数据集上，FreePRM实现了平均F1分数为53.0%，这一成绩优于完全监督训练下的Math-Shepherd PRM模型+24.1%。与开源的其他PRMs相比，FreePRM在性能上也取得了显著优势：相较于RLHFlow-PRM-Mistral-8B（28.4%），其表现高出+24.6%，优于EurusPRM（31.3%）+21.7%，以及Skywork-PRM-7B（42.1%）+10.9%。 这项工作开创了PRMs训练的新范式，显著减少了对昂贵的步骤级注释的依赖，同时保持了强大的性能表现。
URL
https://arxiv.org/abs/2506.03570
