Abstract
3-Dimensional Embodied Reference Understanding (3D-ERU) combines a language description and an accompanying pointing gesture to identify the most relevant target object in a 3D scene. Although prior work has explored pure language-based 3D grounding, there has been limited exploration of 3D-ERU, which also incorporates human pointing gestures. To address this gap, we introduce a data augmentation framework-Imputer, and use it to curate a new benchmark dataset-ImputeRefer for 3D-ERU, by incorporating human pointing gestures into existing 3D scene datasets that only contain language instructions. We also propose Ges3ViG, a novel model for 3D-ERU that achieves ~30% improvement in accuracy as compared to other 3D-ERU models and ~9% compared to other purely language-based 3D grounding models. Our code and dataset are available at this https URL.
Abstract (translated)
三维实体参照理解（3-Dimensional Embodied Reference Understanding，简称3D-ERU）结合了语言描述和伴随的手势指向动作来在三维场景中识别最相关的对象。尽管先前的工作已经探讨了纯基于语言的三维定位问题，但对于整合人类手势指向动作的3D-ERU研究还相对较少。为了解决这一缺口，我们引入了一个数据增强框架——Imputer，并利用它结合现有的仅包含语言指令的三维场景数据集中的手势指向动作，创建了一个新的基准数据集——ImputeRefer，用于促进对3D-ERU的研究。此外，我们提出了一种名为Ges3ViG的新模型，在3D-ERU任务中相较于其他方法提高了约30%的准确率，并且相比于纯基于语言的三维定位模型也提升了约9%的表现。 我们的代码和数据集可在以下链接获取：[提供URL的地方]
URL
https://arxiv.org/abs/2504.09623
