Abstract
A quintessential feature of human intelligence is the ability to create ad hoc conventions over time to achieve shared goals efficiently. We investigate how communication strategies evolve through repeated collaboration as people coordinate on shared procedural abstractions. To this end, we conducted an online unimodal study (n = 98) using natural language to probe abstraction hierarchies. In a follow-up lab study (n = 40), we examined how multimodal communication (speech and gestures) changed during physical collaboration. Pairs used augmented reality to isolate their partner's hand and voice; one participant viewed a 3D virtual tower and sent instructions to the other, who built the physical tower. Participants became faster and more accurate by establishing linguistic and gestural abstractions and using cross-modal redundancy to emphasize key changes from previous interactions. Based on these findings, we extend probabilistic models of convention formation to multimodal settings, capturing shifts in modality preferences. Our findings and model provide building blocks for designing convention-aware intelligent agents situated in the physical world.
Abstract (translated)
人类智能的一个典型特征是随着时间的推移，能够创建临时约定以高效地实现共同目标。我们研究了通过反复合作，沟通策略如何随着人们在共享过程抽象上进行协调而演变。为此，我们开展了一项在线单模态研究（n = 98），使用自然语言来探索抽象层次结构。在后续的实验室研究（n = 40）中，我们考察了多模态通信（包括言语和手势）在物理协作过程中如何变化。实验中，成对参与者通过增强现实技术隔离彼此的手部动作和声音；一名参与者观察一个3D虚拟塔楼，并向另一名正在构建实体塔楼的同伴发出指令。参与者通过建立语言和手势抽象，并利用跨模态冗余来强调先前互动中的关键变化而变得更快且更准确。 基于这些发现，我们将概率模型在约定形成方面的应用扩展到了多模态环境，捕捉到模式偏好转变的趋势。我们的研究结果与模型为设计适用于物理世界的感知常规的智能代理提供了基础构建模块。
URL
https://arxiv.org/abs/2602.08914
