Abstract
To assist humans in open-world environments, robots must interpret ambiguous instructions to locate desired objects. Foundation model-based approaches excel at multimodal grounding, but they lack a principled mechanism for modeling uncertainty in long-horizon tasks. In contrast, Partially Observable Markov Decision Processes (POMDPs) provide a systematic framework for planning under uncertainty but are often limited in supported modalities and rely on restrictive environment assumptions. We introduce LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP), a modular POMDP system that integrates language, gesture, and visual observations for open-world object search. Unlike prior work, LEGS-POMDP explicitly models two sources of partial observability: uncertainty over the target object's identity and its spatial location. In simulation, multimodal fusion significantly outperforms unimodal baselines, achieving an average success rate of 89\% across challenging environments and object categories. Finally, we demonstrate the full system on a quadruped mobile manipulator, where real-world experiments qualitatively validate robust multimodal perception and uncertainty reduction under ambiguous instructions.
Abstract (translated)
为了在开放环境中协助人类，机器人必须能够解析模糊指令来定位所需物体。基于基础模型的方法在多模态语义关联方面表现出色，但在长时间任务中缺乏一种系统的方式来建模不确定性。相比之下，部分可观察马尔科夫决策过程(POMDP)提供了一种在不确定条件下进行规划的系统框架，但通常支持的模态有限，并依赖于严格的环境假设。我们引入了一个模块化POMDP系统——LanguagE and Gesture-Guided Object Search in Partially Observable Environments (LEGS-POMDP)，该系统结合了语言、手势和视觉观察来实现开放世界中的物体搜索功能。 不同于以往的工作，LEGS-POMDP 明确建模了两种部分可观察性的来源：目标对象身份的不确定性及其空间位置的不确定性。在模拟环境中，多模式融合显著优于单模式基线方法，在具有挑战性环境及不同类别对象的情况下，平均成功率为89%。 最后，我们在四足移动机器人上展示了完整的系统，并通过实际实验定性地验证了其强大的多模态感知能力以及在模糊指令下不确定性减少的能力。
URL
https://arxiv.org/abs/2603.04705
