Abstract
Understanding and localizing objects in complex 3D environments from natural language descriptions, known as 3D Visual Grounding (3DVG), is a foundational challenge in embodied AI, with broad implications for robotics, augmented reality, and human-machine interaction. Large-scale pre-trained foundation models have driven significant progress on this front, enabling open-vocabulary 3DVG that allows systems to locate arbitrary objects in a given scene. However, their reliance on pre-trained models constrains 3D perception and reasoning within the inherited knowledge boundaries, resulting in limited generalization to unseen spatial relationships and poor robustness to out-of-distribution scenes. In this paper, we replace this constrained perception with training-free visual and geometric reasoning, thereby unlocking open-world 3DVG that enables the localization of any object in any scene beyond the training data. Specifically, the proposed UniGround operates in two stages: a Global Candidate Filtering stage that constructs scene candidates through training-free 3D topology and multi-view semantic encoding, and a Local Precision Grounding stage that leverages multi-scale visual prompting and structured reasoning to precisely identify the target object. Experiments on ScanRefer and EmbodiedScan show that UniGround achieves 46.1\%/34.1\% Acc@0.25/0.5 on ScanRefer and 28.7\% Acc@0.25 on EmbodiedScan, establishing a new state-of-the-art among zero-shot methods on EmbodiedScan without any 3D supervision. We further evaluate UniGround in real-world environments under uncontrolled reconstruction conditions and substantial domain shift, showing training-free reasoning generalizes robustly beyond curated benchmarks.
Abstract (translated)
从自然语言描述理解并定位复杂三维环境中的物体，即所谓的三维视觉接地（3DVG），是具身人工智能领域的基础挑战之一。这一技术在机器人学、增强现实和人机交互等方面具有广泛的应用前景。大规模预训练的模型在此领域取得了显著进展，使得开放词汇表的3DVG得以实现，允许系统定位给定场景中的任意物体。然而，这些方法依赖于预训练模型，这限制了三维感知与推理在继承知识范围内的灵活性，导致对未见过的空间关系和分布外场景的泛化能力有限且鲁棒性较差。 本文提出了一种新的解决方案：通过无训练的视觉和几何推理来替代这种受限的感知方式。这种方法解锁了一个开放世界的3DVG系统，该系统能够在超出训练数据集范围内的任何场景中定位任意物体。具体而言，所提出的UniGround系统分为两个阶段： 1. 全局候选过滤阶段（Global Candidate Filtering stage）：通过无训练的三维拓扑和多视角语义编码构造场景候选； 2. 局部精准接地阶段（Local Precision Grounding stage）：利用多层次视觉提示和结构化推理精确识别目标物体。 实验结果表明，在ScanRefer和EmbodiedScan数据集上，UniGround在ScanRefer中的Acc@0.25/0.5分别为46.1%/34.1%，在EmbodiedScan中为28.7% Acc@0.25。这使得它在没有三维监督的情况下，成为EmbodiedScan零样本方法的新状态之下的最优性能。此外，在不受控制的重建条件下和显著领域的变化中测试了UniGround的真实世界环境应用，显示出了无训练推理能稳健地超越精心策划的基准测试。
URL
https://arxiv.org/abs/2603.08131
