Abstract
Reinforcement Learning (RL) has become an effective paradigm for enhancing Large Language Models (LLMs) and visual generative models. However, its application in text-to-audio (TTA) generation remains largely under-explored. Prior work typically employs offline methods like Direct Preference Optimization (DPO) and leverages Contrastive Language-Audio Pretraining (CLAP) models as reward functions. In this study, we investigate the integration of online Group Relative Policy Optimization (GRPO) into TTA generation. We adapt the algorithm for Flow Matching-based audio models and demonstrate that online RL significantly outperforms its offline counterparts. Furthermore, we incorporate rewards derived from Large Audio Language Models (LALMs), which can provide fine-grained scoring signals that are better aligned with human perception. With only 470M parameters, our final model, \textbf{Resonate}, establishes a new SOTA on TTA-Bench in terms of both audio quality and semantic alignment.
Abstract (translated)
强化学习（RL）已成为增强大型语言模型（LLMs）和视觉生成模型的有效方法。然而，其在文本到音频（TTA）生成中的应用仍然很大程度上未被探索。先前的研究通常采用离线方法如直接偏好优化（DPO），并利用对比语言-音频预训练（CLAP）模型作为奖励函数。在这项研究中，我们探讨了将在线组相对策略优化（GRPO）集成到TTA生成中的可能性。我们将算法适应于基于流匹配的音频模型，并证明在线RL比其离线方法显著更优。此外，我们还引入了来自大型音频语言模型（LALMs）的奖励机制，这些模型能够提供与人类感知更好地对齐的细粒度评分信号。凭借仅4.7亿参数量，我们的最终模型“Resonate”在TTA-Bench上建立了新的最先进水平，在音频质量和语义一致性方面均表现出色。
URL
https://arxiv.org/abs/2603.11661
