Abstract
Establishing common ground, a shared set of beliefs and mutually recognized facts, is fundamental to collaboration, yet remains a challenge for current AI systems, especially in multimodal, multiparty settings, where the collaborators bring different information to the table. We introduce the Distributed Partial Information Puzzle (DPIP), a collaborative construction task that elicits rich multimodal communication under epistemic asymmetry. We present a multimodal dataset of these interactions, annotated and temporally aligned across speech, gesture, and action modalities to support reasoning over propositional content and belief dynamics. We then evaluate two paradigms for modeling common ground (CG): (1) state-of-the-art large language models (LLMs), prompted to infer shared beliefs from multimodal updates, and (2) an axiomatic pipeline grounded in Dynamic Epistemic Logic (DEL) that incrementally performs the same task. Results on the annotated DPIP data indicate that it poses a challenge to modern LLMs' abilities to track both task progression and belief state.
Abstract (translated)
建立共同基础，即共享的一套信念和双方认可的事实，是合作的基础，但对当前的人工智能系统来说仍然是一个挑战，尤其是在多模态、多方参与的场景中，参与者会带来不同的信息。我们引入了分布式部分信息拼图（Distributed Partial Information Puzzle, DPIP），这是一个在认识论不对称性条件下激发丰富多模态交流的合作构建任务。我们提供了一套此类互动的多模态数据集，并对言语、手势和行动等不同模态进行了标注和时间同步，以支持命题内容和信念动态的推理。接着，我们评估了两种建立共同基础（CG）的方法：(1) 基于最先进的大型语言模型（LLMs），通过提示它们从多模态更新中推断共享信念；以及 (2) 一种基于动态认识逻辑（Dynamic Epistemic Logic, DEL）的公理化流程，该流程逐步完成同样的任务。在标注的DPIP数据上的结果显示，它对现代大型语言模型跟踪任务进展和信念状态的能力提出了挑战。
URL
https://arxiv.org/abs/2603.05450
