Abstract
The goal of agents in multi-agent environments is to maximize total reward against the opposing agents that are encountered. Following a game-theoretic solution concept, such as Nash equilibrium, may obtain a strong performance in some settings; however, such approaches fail to capitalize on historical and observed data from repeated interactions against our opponents. Opponent modeling algorithms integrate machine learning techniques to exploit suboptimal opponents utilizing available data; however, the effectiveness of such approaches in imperfect-information games to date is quite limited. We show that existing opponent modeling approaches fail to satisfy a simple desirable property even against static opponents drawn from a known prior distribution; namely, they do not guarantee that the model approaches the opponent's true strategy even in the limit as the number of game iterations approaches infinity. We develop a new algorithm that is able to achieve this property and runs efficiently by solving a convex minimization problem based on the sequence-form game representation using projected gradient descent. The algorithm is guaranteed to efficiently converge to the opponent's true strategy given observations from gameplay and possibly additional historical data if it is available.
Abstract (translated)
在多代理环境中的代理目标是最大化与遇到的对立代理之间的总奖励。根据博弈论解决方案概念，如纳什均衡，在某些情况下可以获得强大的性能；然而，这类方法未能充分利用反复互动中从对手那里获得的历史和观察数据。对手建模算法通过整合机器学习技术来利用已有的数据对次优对手进行优化；然而，这些方法在不完全信息游戏中迄今为止的有效性相当有限。 我们展示了现有的对手建模方法甚至对于来自已知先验分布的静态对手都无法满足一个简单的理想属性：即它们无法保证模型会随着游戏轮数趋近于无穷时逐渐接近对手的真实策略。为此，我们开发了一种新算法，该算法能够解决这一问题，并通过基于序列形式博弈表示求解凸优化问题（使用投影梯度下降）高效运行。 该算法确保在给定游戏观察数据和可能的额外历史数据的情况下，能有效地收敛到对手的真实策略。
URL
https://arxiv.org/abs/2508.17671
