Abstract
Bridging the gap between complex human instructions and precise 3D object grounding remains a significant challenge in vision and robotics. Existing 3D segmentation methods often struggle to interpret ambiguous, reasoning-based instructions, while 2D vision-language models that excel at such reasoning lack intrinsic 3D spatial understanding. In this paper, we introduce REALM, an innovative MLLM-agent framework that enables open-world reasoning-based segmentation without requiring extensive 3D-specific post-training. We perform segmentation directly on 3D Gaussian Splatting representations, capitalizing on their ability to render photorealistic novel views that are highly suitable for MLLM comprehension. As directly feeding one or more rendered views to the MLLM can lead to high sensitivity to viewpoint selection, we propose a novel Global-to-Local Spatial Grounding strategy. Specifically, multiple global views are first fed into the MLLM agent in parallel for coarse-level localization, aggregating responses to robustly identify the target object. Then, several close-up novel views of the object are synthesized to perform fine-grained local segmentation, yielding accurate and consistent 3D masks. Extensive experiments show that REALM achieves remarkable performance in interpreting both explicit and implicit instructions across LERF, 3D-OVS, and our newly introduced REALM3D benchmarks. Furthermore, our agent framework seamlessly supports a range of 3D interaction tasks, including object removal, replacement, and style transfer, demonstrating its practical utility and versatility. Project page: this https URL.
Abstract (translated)
在视觉和机器人技术领域，将复杂的指令与精确的三维物体定位相结合仍然是一个重大挑战。现有的三维分割方法通常难以解释那些基于推理的模糊指令，而二维视觉语言模型虽然擅长处理此类推理任务，却缺乏对三维空间的理解。为此，在这篇论文中，我们提出了REALM框架，这是一种创新的多模态大规模预训练语言模型（MLLM）与代理相结合的方法，它能够在无需进行大量特定于3D数据的后期训练的情况下执行基于推理的开放世界分割。我们在3D高斯光束表示上直接进行分割，利用其能够渲染非常逼真的新视角的特点，这些视角非常适合MLLM的理解能力。 然而，将一个或多个渲染视图直接输入到MLLM中可能会导致对视点选择的高度敏感性。因此，我们提出了一个新颖的全局到局部空间定位策略：首先并行地向代理中的MLLM提供多个全局视图以进行粗略级别的本地化，并通过聚合这些响应来稳健地识别目标物体；然后，生成该对象的一系列特写的新视角来进行精细分割，从而获得精确且一致的3D掩膜。 经过广泛的实验表明，REALM在理解LERF、3D-OVS以及我们新引入的REALM3D基准测试中的明确和隐含指令方面表现出卓越性能。此外，我们的代理框架还能够无缝支持一系列三维交互任务，包括物体移除、替换及风格转移等操作，这展示了其实用性和灵活性。 项目主页: [链接](this https URL)
URL
https://arxiv.org/abs/2510.16410
