Abstract
Safe Reinforcement Learning (Safe RL) aims to train an RL agent to maximize its performance in real-world environments while adhering to safety constraints, as exceeding safety violation limits can result in severe consequences. In this paper, we propose a novel safe RL approach called Safety Modulated Policy Optimization (SMPO), which enables safe policy function learning within the standard policy optimization framework through safety modulated rewards. In particular, we consider safety violation costs as feedback from the RL environments that are parallel to the standard awards, and introduce a Q-cost function as safety critic to estimate expected future cumulative costs. Then we propose to modulate the rewards using a cost-aware weighting function, which is carefully designed to ensure the safety limits based on the estimation of the safety critic, while maximizing the expected rewards. The policy function and the safety critic are simultaneously learned through gradient descent during online interactions with the environment. We conduct experiments using multiple RL environments and the experimental results demonstrate that our method outperforms several classic and state-of-the-art comparison methods in terms of overall safe RL performance.
Abstract (translated)
安全强化学习（Safe Reinforcement Learning，简称 Safe RL）旨在训练一个强化学习代理在遵守安全性约束的前提下，在现实世界环境中最大化其性能。超出安全违规限制可能会导致严重后果。本文提出了一种名为“安全调制策略优化”（Safety Modulated Policy Optimization，SMPO）的新方法，该方法通过引入基于安全性的奖励调节机制，在标准策略优化框架内实现安全策略函数学习。 具体来说，我们将安全性违规成本视为来自平行于标准奖赏的强化学习环境中的反馈，并引入一个 Q-成本函数作为安全评论家来估计未来的累计预期成本。然后，我们提出使用一种成本感知加权函数对奖励进行调制，该函数经过精心设计，在确保基于安全评论家估算的安全限制的同时，最大化预期收益。 策略函数和安全评论家通过与环境的在线交互期间同时利用梯度下降法学习。我们在多个强化学习环境中进行了实验，并且实验证明了我们的方法在总体安全性强化学习性能方面优于几种经典及最新的比较方法。
URL
https://arxiv.org/abs/2504.03040
