Abstract
Enabling robots to grasp objects specified through natural language is essential for effective human-robot interaction, yet it remains a significant challenge. Existing approaches often struggle with open-form language expressions and typically assume unambiguous target objects without duplicates. Moreover, they frequently rely on costly, dense pixel-wise annotations for both object grounding and grasp configuration. We present Attribute-based Object Grounding and Robotic Grasping (OGRG), a novel framework that interprets open-form language expressions and performs spatial reasoning to ground target objects and predict planar grasp poses, even in scenes containing duplicated object instances. We investigate OGRG in two settings: (1) Referring Grasp Synthesis (RGS) under pixel-wise full supervision, and (2) Referring Grasp Affordance (RGA) using weakly supervised learning with only single-pixel grasp annotations. Key contributions include a bi-directional vision-language fusion module and the integration of depth information to enhance geometric reasoning, improving both grounding and grasping performance. Experiment results show that OGRG outperforms strong baselines in tabletop scenes with diverse spatial language instructions. In RGS, it operates at 17.59 FPS on a single NVIDIA RTX 2080 Ti GPU, enabling potential use in closed-loop or multi-object sequential grasping, while delivering superior grounding and grasp prediction accuracy compared to all the baselines considered. Under the weakly supervised RGA setting, OGRG also surpasses baseline grasp-success rates in both simulation and real-robot trials, underscoring the effectiveness of its spatial reasoning design. Project page: this https URL
Abstract (translated)
使机器人能够通过自然语言抓取指定的物体对于有效的机器人与人交互至关重要，但这一任务仍然面临着重大挑战。现有的方法通常难以处理开放形式的语言表达，并且假定目标物体会很明确且没有重复的对象。此外，它们往往依赖于成本高昂、像素级别的密集标注来为对象定位和抓取配置提供信息。我们提出了基于属性的目标物体定位与机器人抓取（OGRG）这一新颖框架，该框架能够解释开放形式的语言表达，并通过空间推理确定目标物及其平面抓取位置，即使在包含重复实例的场景中也能实现这一点。我们在两种设置下研究了 OGRG：(1) 像素级别监督下的指称抓取合成（RGS），以及 (2) 仅使用单一像素抓取标注进行弱监督学习的指称抓取能力（RGA）。关键贡献包括双向视觉语言融合模块和深度信息集成，以增强几何推理，从而提升定位与抓取性能。实验结果表明，在具有多样空间语言指令的桌面上，OGRG 超过了所有强基线模型的表现。在 RGS 中，它可以在单个 NVIDIA RTX 2080 Ti GPU 上实现每秒17.59帧的操作速度，这使其实现闭环或多物体顺序抓取成为可能，并且其定位和抓取预测准确性超越了所有的基准测试模型。而在弱监督的RGA设置下，OGRG 在模拟和真实机器人实验中也超过了基线的成功抓取率，强调了其空间推理设计的有效性。 项目主页：[此URL](https://this-url.com)
URL
https://arxiv.org/abs/2509.08126
