Abstract
Balancing exploration and exploitation during reinforcement learning fine-tuning of generative models presents a critical challenge, as existing approaches rely on fixed divergence regularization that creates an inherent dilemma: strong regularization preserves model capabilities but limits reward optimization, while weak regularization enables greater alignment but risks instability or reward hacking. We introduce Adaptive Divergence Regularized Policy Optimization (ADRPO), which automatically adjusts regularization strength based on advantage estimates-reducing regularization for high-value samples while applying stronger regularization to poor samples, enabling policies to navigate between exploration and aggressive exploitation according to data quality. Our implementation with Wasserstein-2 regularization for flow matching generative models achieves remarkable results on text-to-image generation, achieving better semantic alignment and diversity than offline methods like DPO and online methods with fixed regularization like ORW-CFM-W2. ADRPO enables a 2B parameter SD3 model to surpass much larger models with 4.8B and 12B parameters in attribute binding, semantic consistency, artistic style transfer, and compositional control while maintaining generation diversity. ADRPO generalizes to KL-regularized fine-tuning of both text-only LLMs and multi-modal reasoning models, enhancing existing online RL methods like GRPO. In LLM fine-tuning, ADRPO demonstrates an emergent ability to escape local optima through active exploration, while in multi-modal audio reasoning, it outperforms GRPO through superior step-by-step reasoning, enabling a 7B model to outperform substantially larger commercial models including Gemini 2.5 Pro and GPT-4o Audio, offering an effective plug-and-play solution to the exploration-exploitation challenge across diverse generative architectures and modalities.
Abstract (translated)
在强化学习（Reinforcement Learning，RL）中对生成模型进行微调时，平衡探索和利用是一个关键挑战。现有的方法依赖于固定的发散正则化（divergence regularization），这种固定的方法产生了固有的两难局面：强烈的正则化可以保持模型的能力但限制了奖励优化的范围；而较弱的正则化虽然能更好地对齐目标，却可能带来不稳定或奖励操纵的风险。为此，我们提出了自适应发散正则化的策略优化（Adaptive Divergence Regularized Policy Optimization, ADRPO），该方法可以根据优势估计自动调整正则化强度——对于高价值样本减少正则化，而对于低质量的样本施加强正则化，从而使得策略能够在探索与积极利用之间根据数据的质量灵活切换。 我们的实现方案中使用了Wasserstein-2正则化的流匹配生成模型，在文本到图像生成的任务上取得了显著成果，其在语义对齐和多样性方面超越了离线方法（如DPO）以及固定正则化在线方法（例如ORW-CFM-W2）。ADRPO允许一个参数量为20亿的SD3模型在属性绑定、语义一致性、艺术风格转换及组合控制等任务上优于具有48亿和120亿参数的更大规模模型，同时保持生成多样性的平衡。 此外，ADRPO适用于KL正则化微调仅限于文本的大语言模型（Large Language Models, LLMs）以及多模态推理模型。它增强了现有在线RL方法（如GRPO）。在LLM微调过程中，ADRPO展示了通过积极探索逃离局部最优解的能力；而在处理多模态音频推理时，相比GRPO，它能凭借更出色的逐步推理能力击败包括Gemini 2.5 Pro和GPT-4o Audio在内的大型商业模型。这使得一个具有70亿参数的模型能够超越这些商业模型，并为各种生成架构及模态中探索与利用之间的挑战提供了一个有效的即插即用解决方案。 综上所述，ADRPO通过自适应调整正则化强度的方式，在多个场景下提升了生成模型的效果和性能，特别是在平衡探索与利用方面表现出色。
URL
https://arxiv.org/abs/2510.18053
