Abstract
As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.
Abstract (translated)
随着机器人进入人类工作空间，它们需要理解具体的人类指令以实现直观流畅的人机交互（HRI）。然而，由于缺乏能够捕捉多样环境中的自然具体互动的大规模数据集，准确的理解变得具有挑战性。现有的数据集存在视角偏差、单一视图收集、非言语手势覆盖不足以及主要关注室内环境等问题。为了解决这些问题，我们推出了Refer360数据集，这是一个大规模的数据集，包含了在室内和室外环境中从多个不同角度采集的具体语言和非语言互动。此外，我们还引入了MuRes，一个多模态引导残差模块，旨在提高具体指示表达的理解能力。MuRes作为一个信息瓶颈，提取出每个模式中显著的信号，并将其强化到预训练表示中，形成互补特征以用于下游任务。 我们在四个HRI数据集上进行了广泛的实验，包括Refer360数据集，并展示了当前多模态模型无法全面捕捉具体互动；然而，在这些模型中加入MuRes可以持续提升性能。这一发现确立了Refer360作为宝贵的基准测试的价值，并表明引导残差学习在提高机器人理解人类环境中的具体指示表达方面具有潜力。
URL
https://arxiv.org/abs/2512.06558
