Abstract
Segmenting objects in an environment is a crucial task for autonomous driving and robotics, as it enables a better understanding of the surroundings of each agent. Although camera sensors provide rich visual details, they are vulnerable to adverse weather conditions. In contrast, radar sensors remain robust under such conditions, but often produce sparse and noisy data. Therefore, a promising approach is to fuse information from both sensors. In this work, we propose a novel framework to enhance camera-only baselines by integrating a diffusion model into a camera-radar fusion architecture. We leverage radar point features to create pseudo-masks using the Segment-Anything model, treating the projected radar points as point prompts. Additionally, we propose a noise reduction unit to denoise these pseudo-masks, which are further used to generate inpainted images that complete the missing information in the original images. Our method improves the camera-only segmentation baseline by 2.63% in mIoU and enhances our camera-radar fusion architecture by 1.48% in mIoU on the Waterscenes dataset. This demonstrates the effectiveness of our approach for semantic segmentation using camera-radar fusion under adverse weather conditions.
Abstract (translated)
在自主驾驶和机器人技术中，对环境中物体进行分割是一项关键任务，这有助于每个代理更好地理解其周围环境。尽管摄像头传感器提供了丰富的视觉细节，但它们容易受到恶劣天气条件的影响。相比之下，雷达传感器在这种条件下仍保持稳定可靠，但通常会产生稀疏且噪声较多的数据。因此，一种有前景的方法是融合来自两种传感器的信息。 在本工作中，我们提出了一种新的框架，通过将扩散模型集成到摄像头-雷达融合架构中来增强仅基于摄像头的基准方法。利用雷达点特征，我们使用Segment-Anything模型创建伪掩码，并将投影后的雷达点视为点提示。此外，我们还提出了一个噪声减少单元，用于净化这些伪掩码，进一步生成修复图像以完成原始图像中的缺失信息。 我们的方法在Waterscenes数据集上提高了仅基于摄像头的分割基准2.63%的mIoU（平均交并比），并且改进了我们的摄像头-雷达融合架构1.48%的mIoU。这证明了我们在恶劣天气条件下使用摄像头-雷达融合进行语义分割方法的有效性。
URL
https://arxiv.org/abs/2505.03679
