Abstract
Deep reinforcement learning has shown remarkable success in continuous control tasks, yet often requires extensive training data, struggles with complex, long-horizon planning, and fails to maintain safety constraints during operation. Meanwhile, Model Predictive Control (MPC) offers explainability and constraint satisfaction, but typically yields only locally optimal solutions and demands careful cost function design. This paper introduces the Q-guided STein variational model predictive Actor-Critic (Q-STAC), a novel framework that bridges these approaches by integrating Bayesian MPC with actor-critic reinforcement learning through constrained Stein Variational Gradient Descent (SVGD). Our method optimizes control sequences directly using learned Q-values as objectives, eliminating the need for explicit cost function design while leveraging known system dynamics to enhance sample efficiency and ensure control signals remain within safe boundaries. Extensive experiments on 2D navigation and robotic manipulation tasks demonstrate that Q-STAC achieves superior sample efficiency, robustness, and optimality compared to state-of-the-art algorithms, while maintaining the high expressiveness of policy distributions. Experiment videos are available on our website: this https URL
Abstract (translated)
深度强化学习在连续控制任务中表现出显著的成功，但仍需要大量的训练数据，并且在处理复杂、长期的规划问题以及确保操作过程中的安全约束时面临挑战。与此同时，模型预测控制（MPC）提供了可解释性和满足约束条件的能力，但通常只能提供局部最优解，并要求精心设计的成本函数。本文介绍了Q-guided STein变分模型预测Actor-Critic (Q-STAC)框架，这是一种新的方法，通过将贝叶斯MPC与基于演员-评论家的强化学习相结合来弥合这两种方法之间的差距，利用了约束Stein Variational Gradient Descent（SVGD）。 我们的方法直接使用已学得的状态动作价值函数（Q值）作为目标优化控制序列，从而消除了显式成本函数设计的需求，并且利用已知的系统动力学来提高样本效率并确保控制信号保持在安全范围内。在二维导航和机器人操作任务上的广泛实验表明，与最先进的算法相比，Q-STAC实现了更优的样本效率、鲁棒性和最优性，同时保持了策略分布的高度表达能力。 实验视频可在我们的网站上查看：[提供的链接]
URL
https://arxiv.org/abs/2507.06625
