Abstract
Curriculum design for reinforcement learning (RL) can speed up an agent's learning process and help it learn to perform well on complex tasks. However, existing techniques typically require domain-specific hyperparameter tuning, involve expensive optimization procedures for task selection, or are suitable only for specific learning objectives. In this work, we consider curriculum design in contextual multi-task settings where the agent's final performance is measured w.r.t. a target distribution over complex tasks. We base our curriculum design on the Zone of Proximal Development concept, which has proven to be effective in accelerating the learning process of RL agents for uniform distribution over all tasks. We propose a novel curriculum, ProCuRL-Target, that effectively balances the need for selecting tasks that are not too difficult for the agent while progressing the agent's learning toward the target distribution via leveraging task correlations. We theoretically justify the task selection strategy of ProCuRL-Target by analyzing a simple learning setting with REINFORCE learner model. Our experimental results across various domains with challenging target task distributions affirm the effectiveness of our curriculum strategy over state-of-the-art baselines in accelerating the training process of deep RL agents.
Abstract (translated)
强化学习（RL）中，课程设计可以加速智能体的学习过程，并帮助它学会在复杂任务上表现出色。然而，现有的技术通常需要针对特定领域进行超参数调整，涉及昂贵的任务选择优化过程，或者仅适用于特定的学习目标。在这项工作中，我们考虑在上下文多任务环境中进行课程设计，其中智能体的最终性能是相对于复杂任务的某个目标分布进行衡量的。我们基于上下文多任务环境中 Zone of Proximal Development（ZOPD）的概念进行课程设计，该概念已经被证明在加速所有任务上 RL 智能体的学习过程中非常有效。我们提出了一个名为 ProCuRL-Target 的全新课程，通过利用任务关联有效地平衡了选择任务既不至于过于困难，又不至于无法继续学习目标分布的需求。我们通过分析一个简单的学习场景（REINFORCE 学习者模型）来理论证明 ProCuRL-Target 的任务选择策略。我们在各种具有具有挑战性目标任务分布的实验领域中进行实验，证实了我们的课程策略在加速深度 RL 智能体训练过程方面的有效性超过了最先进的基准。
URL
https://arxiv.org/abs/2405.02481
