Abstract
We explore the self-play training procedure of large language models (LLMs) in a two-player adversarial language game called Adversarial Taboo. In this game, an attacker and a defender communicate with respect to a target word only visible to the attacker. The attacker aims to induce the defender to utter the target word unconsciously, while the defender tries to infer the target word from the attacker's utterances. To win the game, both players should have sufficient knowledge about the target word and high-level reasoning ability to infer and express in this information-reserved conversation. Hence, we are curious about whether LLMs' reasoning ability can be further enhanced by Self-Play in this Adversarial language Game (SPAG). With this goal, we let LLMs act as the attacker and play with a copy of itself as the defender on an extensive range of target words. Through reinforcement learning on the game outcomes, we observe that the LLMs' performance uniformly improves on a broad range of reasoning benchmarks. Furthermore, iteratively adopting this self-play process can continuously promote LLM's reasoning ability. The code is at this https URL.
Abstract (translated)
我们在两款对抗性语言游戏《Adversarial Taboo》中探索大型语言模型（LLMs）的自玩训练过程。在这款游戏中，攻击者和防御者仅就一个可见目标词进行通信。攻击者的目标是诱导防御者在不自觉的情况下说出目标词，而防御者则试图从攻击者的陈述中推断出目标词。要获胜，双方都应该具备关于目标词的充分知识以及推断和表达高层次能力。因此，我们对LLMs自玩在对抗性语言游戏中（SPAG）是否可以进一步增强推理能力感到好奇。为实现这一目标，我们让LLMs充当攻击者，在广泛的标有目标词的范围内，使用其自身的副本作为防御者。通过在游戏结果上进行强化学习，我们观察到LLM的表现随着各种推理基准的提高而普遍改善。此外，逐步采用这种自玩过程可以持续地促进LLM的推理能力。代码位于此链接处：
URL
https://arxiv.org/abs/2404.10642
