Abstract
Semantic segmentation across visual modalities such as 3D point clouds and panoramic images remains a challenging task, primarily due to the scarcity of annotated data and the limited adaptability of fixed-label models. In this paper, we present JOPP-3D, an open-vocabulary semantic segmentation framework that jointly leverages panoramic and point cloud data to enable language-driven scene understanding. We convert RGB-D panoramic images into their corresponding tangential perspective images and 3D point clouds, then use these modalities to extract and align foundational vision-language features. This allows natural language querying to generate semantic masks on both input modalities. Experimental evaluation on the Stanford-2D-3D-s and ToF-360 datasets demonstrates the capability of JOPP-3D to produce coherent and semantically meaningful segmentations across panoramic and 3D domains. Our proposed method achieves a significant improvement compared to the SOTA in open and closed vocabulary 2D and 3D semantic segmentation.
Abstract (translated)
在视觉模态（如三维点云和全景图像）上进行语义分割仍然是一个具有挑战性的任务，主要原因是标注数据的稀缺以及固定标签模型适应能力有限。本文介绍了 JOPP-3D，这是一种开放式词汇语义分割框架，通过共同利用全景图和点云数据来实现基于语言的场景理解。我们把 RGB-D 全景图像转换为其相应的切线视角图像和 3D 点云，然后使用这些模态提取并对齐基础视觉-语言特征。这使得自然语言查询能够在这两种输入模式上生成语义蒙版。在斯坦福2D-3D-s 和 ToF-360 数据集上的实验评估证明了 JOPP-3D 能够在全景和 3D 域中生成一致且具有语义意义的分割结果。我们提出的方法在开放性和封闭性词汇的 2D 和 3D 语义分割方面相比现有最佳方法取得了显著的进步。
URL
https://arxiv.org/abs/2603.06168
