Abstract
Model-free reinforcement learning methods lack an inherent mechanism to impose behavioural constraints on the trained policies. While certain extensions exist, they remain limited to specific types of constraints, such as value constraints with additional reward signals or visitation density constraints. In this work we try to unify these existing techniques and bridge the gap with classical optimization and control theory, using a generic primal-dual framework for value-based and actor-critic reinforcement learning methods. The obtained dual formulations turn out to be especially useful for imposing additional constraints on the learned policy, as an intrinsic relationship between such dual constraints (or regularization terms) and reward modifications in the primal is reveiled. Furthermore, using this framework, we are able to introduce some novel types of constraints, allowing to impose bounds on the policy's action density or on costs associated with transitions between consecutive states and actions. From the adjusted primal-dual optimization problems, a practical algorithm is derived that supports various combinations of policy constraints that are automatically handled throughout training using trainable reward modifications. The resulting $\texttt{DualCRL}$ method is examined in more detail and evaluated under different (combinations of) constraints on two interpretable environments. The results highlight the efficacy of the method, which ultimately provides the designer of such systems with a versatile toolbox of possible policy constraints.
Abstract (translated)
模型无关强化学习方法缺乏对训练后策略施加行为约束的固有机制。虽然存在某些扩展，但它们仍然局限于特定的约束类型，例如带有额外奖励信号的价值约束或访问密度约束。在这项工作中，我们试图统一这些现有技术，并使用基于价值的actor-critic强化学习方法的泛化二次框架来弥合与经典优化和控制理论之间的差距。所得到的双重形式展开在很大程度上有助于对学习到的策略施加额外的约束，因为这种双重约束（或 regularization 项）与原初在值上的约束之间揭示了一种固有的关系。此外，利用这种框架，我们能够引入一些新颖的约束类型，使得能够对策略的动作密度或连续状态和动作之间的转移成本施加限制。从调整后的原初-二次优化问题中，我们得到了一个实际算法，它在训练过程中自动处理各种策略约束。利用不同的约束组合，对两个可解释的环境进行了评估。结果表明，该方法非常有效，最终为设计这种系统的设计者提供了一个丰富的策略约束工具箱。
URL
https://arxiv.org/abs/2404.16468
