Abstract
Semantic segmentation of 3D LiDAR point clouds is important in urban remote sensing for understanding real-world street environments. This task, by projecting LiDAR point clouds and 3D semantic labels as sparse maps, can be reformulated as a 2D problem. However, the intrinsic sparsity of the projected LiDAR and label maps can result in sparse and inaccurate intermediate 2D semantic predictions, which in return limits the final 3D accuracy. To address this issue, we enhance this task by shaping dense and accurate 2D predictions. Specifically, we develop a multi-modal segmentation model, MM2D3D. By leveraging camera images as auxiliary data, we introduce cross-modal guided filtering to overcome label map sparsity by constraining intermediate 2D semantic predictions with dense semantic relations derived from the camera images; and we introduce dynamic cross pseudo supervision to overcome LiDAR map sparsity by encouraging the 2D predictions to emulate the dense distribution of the semantic predictions from the camera images. Experiments show that our techniques enable our model to achieve intermediate 2D semantic predictions with dense distribution and higher accuracy, which effectively enhances the final 3D accuracy. Comparisons with previous methods demonstrate our superior performance in both 2D and 3D spaces.
Abstract (translated)
三维激光雷达点云的语义分割在城市遥感中对于理解现实世界街道环境非常重要。通过将激光雷达点云和三维语义标签投影为稀疏地图，这一任务可以被重新表述为一个二维问题。然而，投影后的激光雷达图和标签图所固有的稀疏性会导致中间二维语义预测既稀疏又不准确，进而限制了最终的三维精度。 为了应对这个问题，我们通过塑造密集且准确的二维预测来增强此任务。具体而言，我们开发了一种多模态分割模型MM2D3D。利用摄像机图像作为辅助数据，我们引入了跨模态引导过滤器，以克服标签图稀疏性的问题，通过用从摄像机图像中衍生出的密集语义关系约束中间二维语义预测来实现这一目标；同时，我们还引入了动态交叉伪监督，通过鼓励二维预测模仿来自摄像机图像的语义预测的密集分布，从而解决激光雷达地图稀疏性问题。 实验表明，我们的技术使模型能够获得具有密集分布和更高准确性的中间二维语义预测，这有效地提高了最终的三维精度。与先前的方法相比，我们在二维和三维空间中的性能均表现出优越性。
URL
https://arxiv.org/abs/2602.18869
