Abstract
Recent advancements in large language models (LLMs) have been driven by their emergent reasoning capabilities, particularly through long chain-of-thought (CoT) prompting, which enables thorough exploration and deliberation. Despite these advances, long-CoT LLMs often exhibit suboptimal reasoning behaviors, such as overthinking and excessively protracted reasoning chains, which can impair performance. In this paper, we analyze reasoning processes through an optimization lens, framing CoT as a gradient descent procedure where each reasoning step constitutes an update toward problem resolution. Building on this perspective, we introduce RePro (Rectifying Process-level Reward), a novel approach to refine LLM reasoning during post-training. RePro defines a surrogate objective function to assess the optimization process underlying CoT, utilizing a dual scoring mechanism to quantify its intensity and stability. These scores are aggregated into a composite process-level reward, seamlessly integrated into reinforcement learning with verifiable rewards (RLVR) pipelines to optimize LLMs. Extensive experiments across multiple reinforcement learning algorithms and diverse LLMs, evaluated on benchmarks spanning mathematics, science, and coding, demonstrate that RePro consistently enhances reasoning performance and mitigates suboptimal reasoning behaviors.
Abstract (translated)
最近，大型语言模型（LLM）的进步主要得益于其通过长链条思考（CoT）提示所展现的新兴推理能力。尽管取得了这些进展，但使用长链式CoT的LLMs经常表现出次优的推理行为，例如过度思考和过于冗长的推理链，这可能会损害性能。在本文中，我们从优化的角度分析了推理过程，并将CoT视为一种梯度下降程序，在这种程序中，每个推理步骤都是向解决问题方向更新的一部分。在此基础上，我们引入了一种新的方法RePro（修正过程级奖励），该方法通过后训练来精炼LLM的推理能力。RePro定义了一个替代目标函数，用于评估CoT背后优化过程的质量，并使用一种双重评分机制来量化其强度和稳定性。这些分数被整合为一个综合的过程级奖励，并无缝集成到具有可验证回报（RLVR）管道中的强化学习过程中，以优化LLM。通过在多种强化学习算法及不同LLMs上进行的广泛实验，并且在数学、科学和编码等领域基准测试中进行了评估，我们发现RePro持续提高了推理性能并减轻了次优行为。
URL
https://arxiv.org/abs/2512.01925
