Abstract
Multi-agent reinforcement learning (MARL) algorithms often struggle to find strategies close to Pareto optimal Nash Equilibrium, owing largely to the lack of efficient exploration. The problem is exacerbated in sparse-reward settings, caused by the larger variance exhibited in policy learning. This paper introduces MESA, a novel meta-exploration method for cooperative multi-agent learning. It learns to explore by first identifying the agents' high-rewarding joint state-action subspace from training tasks and then learning a set of diverse exploration policies to "cover" the subspace. These trained exploration policies can be integrated with any off-policy MARL algorithm for test-time tasks. We first showcase MESA's advantage in a multi-step matrix game. Furthermore, experiments show that with learned exploration policies, MESA achieves significantly better performance in sparse-reward tasks in several multi-agent particle environments and multi-agent MuJoCo environments, and exhibits the ability to generalize to more challenging tasks at test time.
Abstract (translated)
多元智能体强化学习（MARL）算法通常很难找到接近帕累托最优纳什均衡的战略，这很大程度上是因为缺乏有效的探索。在稀疏奖励环境中，问题进一步加剧，由于策略学习表现出更大的方差。本文介绍了一种名为MESA的新协作多智能体学习元探索方法。它通过首先从训练任务中确定代理器的局部高奖励状态-动作子空间，然后学习一系列多样化的探索策略来“覆盖”该子空间。这些训练探索策略可以与任何离散的MARL算法在测试时间任务中集成。我们首先展示了MESA在多级矩阵游戏中的优势。此外，实验结果表明，在稀疏奖励任务中，MESA在多个多智能体粒子环境和多智能体MuJoCo环境中取得了显著的更好的性能，并且具有在测试时间将任务泛化的能力。
URL
https://arxiv.org/abs/2405.00902
