Abstract
Ensuring the safety of Reinforcement Learning (RL) is crucial for its deployment in real-world applications. Nevertheless, managing the trade-off between reward and safety during exploration presents a significant challenge. Improving reward performance through policy adjustments may adversely affect safety performance. In this study, we aim to address this conflicting relation by leveraging the theory of gradient manipulation. Initially, we analyze the conflict between reward and safety gradients. Subsequently, we tackle the balance between reward and safety optimization by proposing a soft switching policy optimization method, for which we provide convergence analysis. Based on our theoretical examination, we provide a safe RL framework to overcome the aforementioned challenge, and we develop a Safety-MuJoCo Benchmark to assess the performance of safe RL algorithms. Finally, we evaluate the effectiveness of our method on the Safety-MuJoCo Benchmark and a popular safe benchmark, Omnisafe. Experimental results demonstrate that our algorithms outperform several state-of-the-art baselines in terms of balancing reward and safety optimization.
Abstract (translated)
确保强化学习（RL）的安全性对于其在现实应用中的部署至关重要。然而，在探索过程中管理奖励和安全之间的权衡是一个具有挑战性的问题。通过调整策略来提高奖励性能可能会对安全性造成不利影响。在这项研究中，我们旨在通过利用梯度操纵理论来解决这种矛盾关系。首先，我们分析了奖励和安全梯度之间的冲突。接着，我们通过提出软切换策略优化方法来解决奖励和安全优化之间的平衡，并为该方法提供了收敛分析。根据我们的理论审查，我们提供了一个安全的RL框架来克服前述挑战，并开发了一个Safety-MuJoCo基准来评估安全RL算法的性能。最后，我们在Safety-MuJoCo基准和流行的安全基准Omnisafe上评估了我们方法的有效性。实验结果表明，我们的算法在平衡奖励和安全优化方面优于多个最先进的基线。
URL
https://arxiv.org/abs/2405.01677
