Abstract
Current point cloud semantic segmentation has achieved great advances when given sufficient labels. However, the dense annotation of LiDAR point clouds remains prohibitively expensive and time-consuming, unable to keep up with the continuously growing volume of data. In this paper, we propose annotating images with scattered points, followed by utilizing SAM (a Foundation model) to generate semantic segmentation labels for the images. Finally, by mapping the segmentation labels of the images to the LiDAR space using the intrinsic and extrinsic parameters of the camera and LiDAR, we obtain labels for point cloud semantic segmentation, and release Scatter-KITTI and Scatter-nuScenes, which are the first works to utilize image segmentation-based SAM for weakly supervised point cloud semantic segmentation. Furthermore, to mitigate the influence of erroneous pseudo labels obtained from sparse annotations on point cloud features, we propose a multi-modal weakly supervised network for LiDAR semantic segmentation, called MM-ScatterNet. This network combines features from both point cloud and image modalities, enhancing the representation learning of point clouds by introducing consistency constraints between multi-modal features and point cloud features. On the SemanticKITTI dataset, we achieve 66\% of fully supervised performance using only 0.02% of annotated data, and on the NuScenes dataset, we achieve 95% of fully supervised performance using only 0.1% labeled points.
Abstract (translated)
当前的点云语义分割在给出充分标签时取得了很大的进展。然而，对激光雷达点云的密集标注仍然过于昂贵和耗时，无法跟上数据不断增长的数量。在本文中，我们提出使用散射点对图像进行标注，然后利用SAM（一个基础模型）对图像进行语义分割标签生成。最后，通过将图像的语义分割标签映射到激光雷达空间中的内、外参数，我们获得了点云语义分割标签，并释放了Scatter-KITTI和Scatter-nuScenes，这是第一个利用基于图像分割的SAM进行弱监督点云语义分割的工作。此外，为了减轻从稀疏标注中获得的错误伪标签对点云特征的影响，我们提出了一个多模态弱监督网络，称为MM-ScatterNet。该网络结合了点云和图像模态的特征，通过引入多模态特征与点云特征之间的一致性约束，增强了点云的表示学习。在SemanticKITTI数据集上，我们实现了66%的完全监督性能，只需要0.02%的注释数据，而在NuScenes数据集上，我们实现了95%的完全监督性能，只需要0.1%的标注点。
URL
https://arxiv.org/abs/2404.12861
