Abstract
Learning models of the environment from pure interaction is often considered an essential component of building lifelong reinforcement learning agents. However, the common practice in model-based reinforcement learning is to learn models that model every aspect of the agent's environment, regardless of whether they are important in coming up with optimal decisions or not. In this paper, we argue that such models are not particularly well-suited for performing scalable and robust planning in lifelong reinforcement learning scenarios and we propose new kinds of models that only model the relevant aspects of the environment, which we call "minimal value-equivalent partial models". After providing a formal definition for these models, we provide theoretical results demonstrating the scalability advantages of performing planning with such models and then perform experiments to empirically illustrate our theoretical results. Then, we provide some useful heuristics on how to learn these kinds of models with deep learning architectures and empirically demonstrate that models learned in such a way can allow for performing planning that is robust to distribution shifts and compounding model errors. Overall, both our theoretical and empirical results suggest that minimal value-equivalent partial models can provide significant benefits to performing scalable and robust planning in lifelong reinforcement learning scenarios.
Abstract (translated)
从纯交互中学习环境模型通常被视为构建长期强化学习代理的重要组件。然而,基于模型的强化学习常见的做法是学习模型,以模拟代理环境的所有方面,无论这些方面是否在做出最佳决策中非常重要。在本文中,我们认为这些模型在长期强化学习场景下进行可扩展和稳健规划并不特别合适,因此我们提出了新的种类模型,仅模拟环境的相关方面,我们称之为“最小价值等价部分模型”。在为这些模型提供正式定义后,我们提供了理论结果,证明使用这些模型进行规划的可扩展优势,然后进行实验以Empirically 证明我们的理论结果。随后,我们提供了一些有用的启发式,关于如何使用深度学习架构学习这些类型模型,并Empirically 证明,以这种方式学习模型可以允许进行稳健到分布 shift 和模型错误累加的规划。总体而言,我们的理论和实验结果都表明,最小价值等价部分模型可以为长期强化学习场景下的可扩展和稳健规划提供重大好处。
URL
https://arxiv.org/abs/2301.10119
