Abstract
Model-Agnostic Meta-Learning (MAML) is a versatile meta-learning framework applicable to both supervised learning and reinforcement learning (RL). However, applying MAML to meta-reinforcement learning (meta-RL) presents notable challenges. First, MAML relies on second-order gradient computations, leading to significant computational and memory overhead. Second, the nested structure of optimization increases the problem's complexity, making convergence to a global optimum more challenging. To overcome these limitations, we propose Directed-MAML, a novel task-directed meta-RL algorithm. Before the second-order gradient step, Directed-MAML applies an additional first-order task-directed approximation to estimate the effect of second-order gradients, thereby accelerating convergence to the optimum and reducing computational cost. Experimental results demonstrate that Directed-MAML surpasses MAML-based baselines in computational efficiency and convergence speed in the scenarios of CartPole-v1, LunarLander-v2 and two-vehicle intersection crossing. Furthermore, we show that task-directed approximation can be effectively integrated into other meta-learning algorithms, such as First-Order Model-Agnostic Meta-Learning (FOMAML) and Meta Stochastic Gradient Descent(Meta-SGD), yielding improved computational efficiency and convergence speed.
Abstract (translated)
模型无关元学习（Model-Agnostic Meta-Learning，MAML）是一个适用于监督学习和强化学习（RL）的灵活元学习框架。然而，在将MAML应用于元强化学习（meta-RL）时会遇到显著挑战。首先，MAML依赖于二阶导数计算，这会导致较大的计算量和内存开销。其次，优化过程中的嵌套结构增加了问题复杂性，使得达到全局最优解更加困难。 为了克服这些限制，我们提出了Directed-MAML，一种新颖的任务导向型元强化学习算法。在进行二阶梯度步骤之前，Directed-MAML应用一个额外的一阶任务导向近似来估计二阶导数的影响，从而加速收敛到最佳点并减少计算成本。实验结果表明，在CartPole-v1、LunarLander-v2及两车交叉路口等场景中，Directed-MAML在计算效率和收敛速度方面均超越了基于MAML的基准算法。 此外，我们还展示了任务导向近似可以有效地集成到其他元学习算法中，例如一阶模型无关元学习（First-Order Model-Agnostic Meta-Learning，FOMAML）和元随机梯度下降（Meta Stochastic Gradient Descent，Meta-SGD），从而提高计算效率并加快收敛速度。
URL
https://arxiv.org/abs/2510.00212
