Abstract
The commonly used Reinforcement Learning (RL) model, MDPs (Markov Decision Processes), has a basic premise that rewards depend on the current state and action only. However, many real-world tasks are non-Markovian, which has long-term memory and dependency. The reward sparseness problem is further amplified in non-Markovian scenarios. Hence learning a non-Markovian task (NMT) is inherently more difficult than learning a Markovian one. In this paper, we propose a novel \textbf{Par}allel and \textbf{Mod}ular RL framework, ParMod, specifically for learning NMTs specified by temporal logic. With the aid of formal techniques, the NMT is modulaized into a series of sub-tasks based on the automaton structure (equivalent to its temporal logic counterpart). On this basis, sub-tasks will be trained by a group of agents in a parallel fashion, with one agent handling one sub-task. Besides parallel training, the core of ParMod lies in: a flexible classification method for modularizing the NMT, and an effective reward shaping method for improving the sample efficiency. A comprehensive evaluation is conducted on several challenging benchmark problems with respect to various metrics. The experimental results show that ParMod achieves superior performance over other relevant studies. Our work thus provides a good synergy among RL, NMT and temporal logic.
Abstract (translated)
常用的强化学习（RL）模型——马尔可夫决策过程（MDPs），其基本前提是奖励仅依赖于当前状态和动作。然而，许多现实世界任务是非马尔可夫的，具有长期记忆和依赖性。在非马尔可夫场景中，稀疏奖励问题进一步加剧。因此，学习一个非马尔可夫任务（NMT）本质上比学习一个马尔可夫任务更加困难。本文提出了一种新颖的**并行与模块化RL框架**——ParMod，专门用于基于时态逻辑的学习NMTs。借助形式技术，根据自动机结构（相当于其对应的时态逻辑），将NMT分解为一系列子任务。在此基础上，一组代理将以并行的方式训练这些子任务，每个代理处理一个子任务。除了并行训练外，ParMod的核心在于：一种灵活的分类方法用于模块化NMT，以及一种有效的奖励塑形方法以提高采样效率。在多个具有挑战性的基准问题上进行了全面评估，并采用了各种度量标准。实验结果表明，ParMod的表现优于其他相关研究。因此，我们的工作为RL、NMT与时态逻辑提供了一个良好的协同作用。
URL
https://arxiv.org/abs/2412.12700
