Abstract
Multi-agent reinforcement learning (MARL) is increasingly used to design learning-enabled agents that interact in shared environments. However, training MARL algorithms in general-sum games remains challenging: learning dynamics can become unstable, and convergence guarantees typically hold only in restricted settings such as two-player zero-sum or fully cooperative games. Moreover, when agents have heterogeneous and potentially conflicting preferences, it is unclear what system-level objective should guide learning. In this paper, we propose a new MARL pipeline called Near-Potential Policy Optimization (NePPO) for computing approximate Nash equilibria in mixed cooperative--competitive environments. The core idea is to learn a player-independent potential function such that the Nash equilibrium of a cooperative game with this potential as the common utility approximates a Nash equilibrium of the original game. To this end, we introduce a novel MARL objective such that minimizing this objective yields the best possible potential function candidate and consequently an approximate Nash equilibrium of the original game. We develop an algorithmic pipeline that minimizes this objective using zeroth-order gradient descent and returns an approximate Nash equilibrium policy. We empirically show the superior performance of this approach compared to popular baselines such as MAPPO, IPPO and MADDPG.
Abstract (translated)
多智能体强化学习（MARL）被广泛用于设计能够在共享环境中互动的学习型代理。然而，在总和博弈中训练MARL算法仍然存在挑战：学习动态可能会变得不稳定，且收敛保证通常仅在受限情况下有效，如两人零和游戏或完全合作的游戏。此外，当代理具有异质且可能冲突的偏好时，尚不清楚应使用何种系统级目标来指导学习过程。 本文提出了一种新的MARL流水线——近似势函数策略优化（NePPO），用于计算混合合作-竞争环境中的近似纳什均衡。核心思想是学习一种独立于玩家的潜在函数，使得具有该潜在函数作为公共效用的合作博弈的纳什均衡能够接近原始博弈的近似纳什均衡。 为此，我们引入了一种新的MARL目标函数，通过最小化这个目标函数可以获得最佳的势函数候选，并进而得出原游戏的一个近似纳什均衡策略。我们开发了一个算法流水线，使用零阶梯度下降来最小化该目标函数，并返回一个近似纳什均衡策略。 实验结果表明，与流行的基准方法（如MAPPO、IPPO和MADDPG）相比，这种方法在性能上表现出显著优势。
URL
https://arxiv.org/abs/2603.06977
