Abstract
Current frameworks for consensus statement generation with large language models lack the inherent structure needed to provide provable fairness guarantees when aggregating diverse free-form opinions. We model the task as a multi-objective, token-level Markov Decision Process (MDP), where each objective corresponds to an agent's preference. Token-level rewards for each agent are derived from their policy (e.g., a personalized language model). This approach utilizes the finding that such policies implicitly define optimal Q-functions, providing a principled way to quantify rewards at each generation step without a value function (Rafailov et al., 2024). This MDP formulation creates a formal structure amenable to analysis using principles from social choice theory. We propose two approaches grounded in social choice theory. First, we propose a stochastic generation policy guaranteed to be in the ex-ante core, extending core stability concepts from voting theory to text generation. This policy is derived from an underlying distribution over complete statements that maximizes proportional fairness (Nash Welfare). Second, for generating a single statement, we target the maximization of egalitarian welfare using search algorithms within the MDP framework. Empirically, experiments using language models to instantiate agent policies show that search guided by the egalitarian objective generates consensus statements with improved worst-case agent alignment compared to baseline methods, including the Habermas Machine (Tessler et al., 2024).
Abstract (translated)
当前用于大型语言模型共识声明生成的框架缺乏内在结构，无法在聚合不同自由形式的意见时提供可证明的公平性保证。我们把这个任务建模为一个多目标、基于标记级别的马尔科夫决策过程（MDP），每个目标对应于一个代理人的偏好。每一个代理人的令牌级别奖励是从他们的策略中推导出来的（例如，个性化语言模型）。这种方法利用了这样一个发现：这样的策略隐含地定义了最优的Q函数，提供了一种量化每一步生成过程中奖励的原则方法，而无需价值函数（Rafailov等人，2024年）。这种MDP结构使得应用社会选择理论原理进行分析成为可能。 我们提出了两种基于社会选择理论的方法。首先，我们提出了一种保证在事前核心内的随机生成策略，将投票理论中的核心稳定性概念扩展到文本生成中。该策略来源于最大化比例公平性的完整声明的基础分布（纳什福利）。其次，在生成单一声明时，我们在MDP框架内使用搜索算法来追求平等福利的最大化。 实验结果表明，当用语言模型实现代理人策略时，基于平等目标引导的搜索可以产生比基线方法更优的最差情况代理对齐共识声明，包括哈贝马斯机器（Tessler等人，2024年）。
URL
https://arxiv.org/abs/2510.14106
