Abstract
Reinforcement learning is a popular method of finding optimal solutions to complex problems. Algorithms like Q-learning excel at learning to solve stochastic problems without a model of their environment. However, they take longer to solve deterministic problems than is necessary. Q-learning can be improved to better solve deterministic problems by introducing such a model-based approach. This paper introduces the recursive backwards Q-learning (RBQL) agent, which explores and builds a model of the environment. After reaching a terminal state, it recursively propagates its value backwards through this model. This lets each state be evaluated to its optimal value without a lengthy learning process. In the example of finding the shortest path through a maze, this agent greatly outperforms a regular Q-learning agent.
Abstract (translated)
强化学习是一种解决复杂问题最优解的流行方法。像Q-学习这样的算法在解决没有环境模型的随机问题方面表现出色。然而，它们解决确定性问题的时间比必要的要长。通过引入基于模型的方法，可以提高Q-学习的确定性问题的解决能力。本文介绍了递归后反向Q学习（RBQL）代理，它探索并构建环境的模型。在达到终端状态后，它通过这个模型递归地传播其值。这让每个状态都能在没有长篇累赘的学习过程中评估到最优值。在找寻迷宫中最短路径的例子中，这个代理大大超过了普通Q-学习代理的表现。
URL
https://arxiv.org/abs/2404.15822
