Abstract
LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data remains limited compared to RGB datasets with millions of samples. We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds, and leverage self-supervised 3D representations for state-of-the-art results. Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality; (ii) transfer knowledge from large-scale image-pretrained generative models to LiDAR generation, mitigating limited LiDAR datasets; and (iii) enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model. On the KITTI-360 benchmark R3DPA achieves state of the art performance. Code and pretrained models are available at this https URL.
Abstract (translated)
LiDAR场景合成是解决机器人任务（如自动驾驶）中三维数据稀缺问题的一种新兴解决方案。最近的方法使用扩散模型或流匹配模型来生成逼真的场景，但与包含数百万样本的RGB数据集相比，3D数据仍然非常有限。我们引入了R3DPA，这是第一个利用图像预训练先验解锁LiDAR点云潜力，并采用自我监督三维表示以实现最佳性能的LiDAR场景生成方法。具体来说： 1. 我们将生成模型的中间特征与自监督三维特征对齐，这显著提高了生成质量。 2. 将大规模图像预训练生成模型的知识转移到LiDAR生成中，从而缓解了有限的LiDAR数据集问题。 3. 仅使用无条件模型就能够在推理过程中实现点云控制，进行对象修复和场景混合。 在KITTI-360基准测试上，R3DPA实现了最佳性能。代码和预训练模型可在提供的链接中获得。
URL
https://arxiv.org/abs/2601.07692
