Abstract
We present Splat-MOVER, a modular robotics stack for open-vocabulary robotic manipulation, which leverages the editability of Gaussian Splatting (GSplat) scene representations to enable multi-stage manipulation tasks. Splat-MOVER consists of: (i) $\textit{ASK-Splat}$, a GSplat representation that distills latent codes for language semantics and grasp affordance into the 3D scene. ASK-Splat enables geometric, semantic, and affordance understanding of 3D scenes, which is critical for many robotics tasks; (ii) $\textit{SEE-Splat}$, a real-time scene-editing module using 3D semantic masking and infilling to visualize the motions of objects that result from robot interactions in the real-world. SEE-Splat creates a "digital twin" of the evolving environment throughout the manipulation task; and (iii) $\textit{Grasp-Splat}$, a grasp generation module that uses ASK-Splat and SEE-Splat to propose candidate grasps for open-world objects. ASK-Splat is trained in real-time from RGB images in a brief scanning phase prior to operation, while SEE-Splat and Grasp-Splat run in real-time during operation. We demonstrate the superior performance of Splat-MOVER in hardware experiments on a Kinova robot compared to two recent baselines in four single-stage, open-vocabulary manipulation tasks, as well as in four multi-stage manipulation tasks using the edited scene to reflect scene changes due to prior manipulation stages, which is not possible with the existing baselines. Code for this project and a link to the project page will be made available soon.
Abstract (translated)
我们提出了Splat-MOVER,一个开放词汇机器人操作的模块化机器人技术堆栈,利用Gaussian Splatting(GSplat)场景表示的编辑性,实现多阶段操作任务。Splat-MOVER由以下三个部分组成: (i) ASK-Splat,一种GSplat表示,用于将语言语义和抓握可得性从3D场景中提取出来。ASK-Splat能够实现对3D场景的几何、语义和抓握理解,这对许多机器人任务非常重要; (ii) SEE-Splat,一种实时场景编辑模块,使用3D语义掩码和填充来绘制机器人交互在现实世界中的物体运动;SEE-Splat创建了一个随操作任务不断演变的环境的“数字双胞胎”; (iii) Grasp-Splat,一种用于生成抓握的模块,使用ASK-Splat和SEE-Splat来提出开放世界对象的抓握候选者。ASK-Splat在操作之前对RGB图像进行实时训练,而SEE-Splat和Grasp-Splat在操作期间实时运行。我们证明了Splat-MOVER在Kinova机器人上的硬件实验中相对于两个最近的基本模型在四个单阶段、开放词汇操作任务中的卓越性能,以及在四个多阶段操作任务中使用编辑场景反映先前的操作阶段,这是目前的基本模型所不能实现的。该项目的代码和对项目的链接将很快提供。
URL
https://arxiv.org/abs/2405.04378
