Abstract
In natural language processing tasks, pure reinforcement learning (RL) fine-tuning methods often suffer from inefficient exploration and slow convergence; while supervised fine-tuning (SFT) methods, although efficient in training, have limited performance ceiling and less solid theoretical foundation compared to RL. To address efficiency-capability trade-off, we propose the Guess-Think-Answer (GTA) framework that combines the efficiency of SFT with the capability gains of RL in a unified training paradigm. GTA works by having the model first produce a provisional guess (optimized via cross-entropy loss), then reflect on this guess before generating the final answer, with RL rewards shaping both the final output and the format of the entire GTA structure. This hybrid approach achieves both faster convergence than pure RL and higher performance ceiling than pure SFT. To mitigate gradient conflicts between the two training signals, we employ loss masking and gradient constraints. Empirical results on four text classification benchmarks demonstrate that GTA substantially accelerates convergence while outperforming both standalone SFT and RL baselines.
Abstract (translated)
在自然语言处理任务中，纯粹的强化学习（RL）微调方法常常因为探索效率低下和收敛速度慢而面临挑战；而监督微调（SFT）方法虽然训练效率高，但其性能上限较低，并且与RL相比理论基础较弱。为了解决效率与能力之间的权衡问题，我们提出了Guess-Think-Answer (GTA) 框架，该框架结合了SFT的高效性和RL的能力提升，在统一的训练范式中实现了两者的优点。GTA的工作机制是让模型首先生成一个初步猜测（通过交叉熵损失进行优化），然后基于这个初步猜测进行反思，最终输出答案，其中强化学习奖励不仅影响最终输出，还塑造整个GTA结构的形式。这种混合方法比纯粹的RL收敛更快，并且在性能上限方面优于纯粹的SFT。为了缓解两种训练信号之间的梯度冲突，我们采用了损失掩码和梯度约束技术。实验证明，在四个文本分类基准数据集上，GTA不仅大大加速了收敛速度，还超过了独立使用的SFT和RL基线方法的表现。
URL
https://arxiv.org/abs/2509.12108
