Abstract
Recent studies have shown the great potential of diffusion models in improving reinforcement learning (RL) by modeling complex policies, expressing a high degree of multi-modality, and efficiently handling high-dimensional continuous control tasks. However, there is currently limited research on how to optimize diffusion-based polices (e.g., Diffusion Policy) fast and stably. In this paper, we propose an Adam-based Diffusion Policy Optimization (ADPO), a fast algorithmic framework containing best practices for fine-tuning diffusion-based polices in robotic control tasks using the adaptive gradient descent method in RL. Adaptive gradient method is less studied in training RL, let alone diffusion-based policies. We confirm that ADPO outperforms other diffusion-based RL methods in terms of overall effectiveness for fine-tuning on standard robotic tasks. Concretely, we conduct extensive experiments on standard robotic control tasks to test ADPO, where, particularly, six popular diffusion-based RL methods are provided as benchmark methods. Experimental results show that ADPO acquires better or comparable performance than the baseline methods. Finally, we systematically analyze the sensitivity of multiple hyperparameters in standard robotics tasks, providing guidance for subsequent practical applications. Our video demonstrations are released in this https URL.
Abstract (translated)
最近的研究表明，扩散模型在通过建模复杂的策略、表达高多模态性以及高效处理高维连续控制任务方面，具有显著提升强化学习（RL）的潜力。然而，目前关于如何快速且稳定地优化基于扩散的策略（如Diffusion Policy）的研究还很有限。在这篇论文中，我们提出了一种基于Adam的方法——扩散策略优化(ADPO)，这是一个快速的算法框架，使用自适应梯度下降方法在强化学习中对机器人控制任务中的基于扩散的策略进行微调。自适应梯度法在训练RL中研究较少，更不用说针对基于扩散的政策了。我们确认，在标准机器人任务上的微调总体效果上，ADPO优于其他基于扩散的方法。 具体来说，我们在标准的机器人控制任务上进行了广泛的实验以测试ADPO的有效性，并提供了六种流行的基于扩散的强化学习方法作为基准方法进行对比。实验证明，ADPO在性能上至少达到了与这些基准方法相同的水平，或者更好。最后，我们系统地分析了标准机器人任务中多个超参数的敏感度，为后续的实际应用提供了指导。我们的视频演示可在[此处](https://this.https.URL)查看。
URL
https://arxiv.org/abs/2505.08376
