Abstract
Daily life support robots must interpret ambiguous verbal instructions involving demonstratives such as ``Bring me that cup,'' even when objects or users are out of the robot's view. Existing approaches to exophora resolution primarily rely on visual data and thus fail in real-world scenarios where the object or user is not visible. We propose Multimodal Interactive Exophora resolution with user Localization (MIEL), which is a multimodal exophora resolution framework leveraging sound source localization (SSL), semantic mapping, visual-language models (VLMs), and interactive questioning with GPT-4o. Our approach first constructs a semantic map of the environment and estimates candidate objects from a linguistic query with the user's skeletal data. SSL is utilized to orient the robot toward users who are initially outside its visual field, enabling accurate identification of user gestures and pointing directions. When ambiguities remain, the robot proactively interacts with the user, employing GPT-4o to formulate clarifying questions. Experiments in a real-world environment showed results that were approximately 1.3 times better when the user was visible to the robot and 2.0 times better when the user was not visible to the robot, compared to the methods without SSL and interactive questioning. The project website is this https URL.
Abstract (translated)
日常生活中，支持机器人的任务之一是理解含糊的口头指令，例如“把那个杯子给我”，即使物体或用户不在机器人视野范围内。现有解决外指问题的方法主要依赖于视觉数据，在实际场景中当对象或用户不可见时则无法发挥作用。 为此，我们提出了一种多模态交互式外指解析框架（Multimodal Interactive Exophora resolution with user Localization, MIEL），该框架结合了声音源定位（SSL）、语义地图构建、视觉-语言模型（VLMs）以及与用户互动提问的方法。MIEL首先基于环境信息构建一个语义地图，并使用用户的骨骼数据估计指令中提到的潜在对象候选者。利用SSL，机器人可以转向当前不在其视野范围内的用户方向，从而准确识别手势和指向指示。在存在歧义的情况下，机器人主动与用户进行交互，通过GPT-4o生成澄清问题以获得更多信息。 实验结果显示，在真实环境中，当用户处于可见范围内时，MIEL的性能大约是不使用SSL和互动提问方法的1.3倍；而当用户不可见时，这一提升更是达到了2.0倍。项目的官方网站位于此链接：https URL。
URL
https://arxiv.org/abs/2508.16143
