Abstract
Human-robot interaction is increasingly moving toward multi-robot, socially grounded environments. Existing systems struggle to integrate multimodal perception, embodied expression, and coordinated decision-making in a unified framework. This limits natural and scalable interaction in shared physical spaces. We address this gap by introducing a multimodal framework for human-multi-agent interaction in which each robot operates as an autonomous cognitive agent with integrated multimodal perception and Large Language Model (LLM)-driven planning grounded in embodiment. At the team level, a centralized coordination mechanism regulates turn-taking and agent participation to prevent overlapping speech and conflicting actions. Implemented on two humanoid robots, our framework enables coherent multi-agent interaction through interaction policies that combine speech, gesture, gaze, and locomotion. Representative interaction runs demonstrate coordinated multimodal reasoning across agents and grounded embodied responses. Future work will focus on larger-scale user studies and deeper exploration of socially grounded multi-agent interaction dynamics.
Abstract (translated)
人机交互正日益向多机器人、社会具身化的环境发展。现有系统难以在统一框架内整合多模态感知、具身表达与协调决策，这限制了共享物理空间中的自然且可扩展的交互。我们通过引入一个面向人-多智能体交互的多模态框架来解决这一缺口：每个机器人作为自主认知智能体运行，具备集成多模态感知与具身化的大语言模型（LLM）驱动规划。在团队层面，集中式协调机制管理轮次转换与智能体参与，防止语音重叠与动作冲突。该框架在两台仿人机器人上实现，通过融合语音、手势、注视与移动的交互策略，使多智能体能够开展连贯交互。典型交互实例展示了跨智能体的多模态协调推理与具身化响应。未来工作将聚焦更大规模用户研究，并深入探索社会具身化的多智能体交互动态。
URL
https://arxiv.org/abs/2603.23271
