Abstract
This paper presents a new method for the zero-shot open-vocabulary semantic segmentation (OVSS) of 3D automotive lidar data. To circumvent the recognized image-text modality gap that is intrinsic to approaches based on Vision Language Models (VLMs) such as CLIP, our method relies instead on image generation from text, to create prototype images. Given a 3D network distilled from a 2D Vision Foundation Model (VFM), we then label a point cloud by matching 3D point features with 2D image features of these prototypes. Our method is state-of-the-art for OVSS on nuScenes and SemanticKITTI. Code, pre-trained models, and generated images are available at this https URL.
Abstract (translated)
本文提出了一种针对3D车载激光雷达数据的零样本开放词汇语义分割（OVSS）新方法。为规避基于视觉语言模型（如CLIP）方法中固有的图像-文本模态差异问题，本方法转而依赖从文本生成图像来创建原型图像。给定一个从2D视觉基础模型（VFM）蒸馏得到的3D网络，我们通过将3D点特征与这些原型的2D图像特征进行匹配，从而为点云标注标签。该方法在nuScenes和SemanticKITTI数据集上的OVSS任务中达到最先进水平。代码、预训练模型及生成图像可通过此链接获取。
URL
https://arxiv.org/abs/2604.01361
