Abstract
Diffusion models (DMs) excel in photo-realistic image synthesis, but their adaptation to LiDAR scene generation poses a substantial hurdle. This is primarily because DMs operating in the point space struggle to preserve the curve-like patterns and 3D geometry of LiDAR scenes, which consumes much of their representation power. In this paper, we propose LiDAR Diffusion Models (LiDMs) to generate LiDAR-realistic scenes from a latent space tailored to capture the realism of LiDAR scenes by incorporating geometric priors into the learning pipeline. Our method targets three major desiderata: pattern realism, geometry realism, and object realism. Specifically, we introduce curve-wise compression to simulate real-world LiDAR patterns, point-wise coordinate supervision to learn scene geometry, and patch-wise encoding for a full 3D object context. With these three core designs, our method achieves competitive performance on unconditional LiDAR generation in 64-beam scenario and state of the art on conditional LiDAR generation, while maintaining high efficiency compared to point-based DMs (up to 107$\times$ faster). Furthermore, by compressing LiDAR scenes into a latent space, we enable the controllability of DMs with various conditions such as semantic maps, camera views, and text prompts. Our code and pretrained weights are available at this https URL.
Abstract (translated)
扩散模型（DMs）在照片现实图像合成方面表现出色，但将它们应用于激光雷达（LiDAR）场景生成存在重大挑战。这主要是因为在点空间操作的DM很难保留LiDAR场景的曲线状模式和3D几何，这会消耗它们的代表力。在本文中，我们提出了LiDAR扩散模型（LiDMs）来从针对LiDAR场景的潜在空间中生成LiDAR真实场景，通过将几何先验集成到学习管道中来实现这一目标。我们的方法针对三个主要需求：模式现实、几何现实和物体现实。具体来说，我们引入了曲线级压缩来模拟真实世界LiDAR模式，点级坐标监督来学习场景几何，以及完整的3D对象编码来提供对3D对象的全面信息。 通过这三个核心设计，我们的方法在64束场景下的无条件LiDAR生成上实现了与条件LiDAR生成相匹敌的竞争性能，同时在条件LiDAR生成方面达到了最先进的水平，同时保持了与基于点的DM相比高达107倍的高效率。此外，通过将LiDAR场景压缩到潜在空间中，我们使得DM具有各种条件下的可控性，如语义图、相机视图和文本提示。 我们的代码和预训练权重可以从以下链接获取：https://www.xxx.com/。
URL
https://arxiv.org/abs/2404.00815
