Abstract
Recent advancements in 3D diffusion-based semantic scene generation have gained attention. However, existing methods rely on unconditional generation and require multiple resampling steps when editing scenes, which significantly limits their controllability and flexibility. To this end, we propose SSEditor, a controllable Semantic Scene Editor that can generate specified target categories without multiple-step resampling. SSEditor employs a two-stage diffusion-based framework: (1) a 3D scene autoencoder is trained to obtain latent triplane features, and (2) a mask-conditional diffusion model is trained for customizable 3D semantic scene generation. In the second stage, we introduce a geometric-semantic fusion module that enhance the model's ability to learn geometric and semantic information. This ensures that objects are generated with correct positions, sizes, and categories. Extensive experiments on SemanticKITTI and CarlaSC demonstrate that SSEditor outperforms previous approaches in terms of controllability and flexibility in target generation, as well as the quality of semantic scene generation and reconstruction. More importantly, experiments on the unseen Occ-3D Waymo dataset show that SSEditor is capable of generating novel urban scenes, enabling the rapid construction of 3D scenes.
Abstract (translated)
最近，基于3D扩散的语义场景生成技术引起了广泛关注。然而，现有的方法依赖于无条件生成，并且在编辑场景时需要多次重采样步骤，这极大地限制了它们的可控性和灵活性。为此，我们提出了SSEditor，一个可控制的语义场景编辑器，它可以生成指定的目标类别而无需多步重采样。SSEditor采用了基于扩散的两阶段框架：（1）训练3D场景自编码器以获得潜在的三平面特征；（2）训练带有掩码条件的扩散模型进行定制化的3D语义场景生成。在第二阶段，我们引入了一个几何-语义融合模块，增强了模型学习几何和语义信息的能力。这确保了生成的对象具有正确的位置、大小和类别。在SemanticKITTI和CarlaSC上的广泛实验表明，SSEditor在目标生成的可控性和灵活性方面以及语义场景生成和重建的质量上优于先前的方法。更重要的是，在未见过的Occ-3D Waymo数据集上的实验显示，SSEditor能够生成新颖的城市场景，使3D场景的快速构建成为可能。
URL
https://arxiv.org/abs/2411.12290
