Abstract
We present Spatial Region 3D (SR-3D) aware vision-language model that connects single-view 2D images and multi-view 3D data through a shared visual token space. SR-3D supports flexible region prompting, allowing users to annotate regions with bounding boxes, segmentation masks on any frame, or directly in 3D, without the need for exhaustive multi-frame labeling. We achieve this by enriching 2D visual features with 3D positional embeddings, which allows the 3D model to draw upon strong 2D priors for more accurate spatial reasoning across frames, even when objects of interest do not co-occur within the same view. Extensive experiments on both general 2D vision language and specialized 3D spatial benchmarks demonstrate that SR-3D achieves state-of-the-art performance, underscoring its effectiveness for unifying 2D and 3D representation space on scene understanding. Moreover, we observe applicability to in-the-wild videos without sensory 3D inputs or ground-truth 3D annotations, where SR-3D accurately infers spatial relationships and metric measurements.
Abstract (translated)
我们提出了一种空间区域3D（SR-3D）感知的视觉语言模型，该模型通过共享的视觉令牌空间将单视图2D图像与多视图3D数据连接起来。SR-3D支持灵活的区域提示功能，允许用户使用边界框或分割掩码在任何帧上进行注释，或者直接在3D中进行注释，而无需进行全面的多帧标注。我们通过增强2D视觉特征以利用3D位置嵌入来实现这一点，这使3D模型能够借助强大的2D先验知识进行更准确的空间推理，即使感兴趣的物体不同时出现在同一视图内也是如此。在通用2D视觉语言和专门的3D空间基准测试上的广泛实验表明，SR-3D达到了最先进的性能水平，证明了其在统一2D和3D表示空间方面对场景理解的有效性。此外，我们观察到SR-3D适用于没有感官3D输入或地面真实3D注释的真实世界视频中，在这些情况下，SR-3D能够准确地推断出空间关系和度量测量值。
URL
https://arxiv.org/abs/2509.13317
