Abstract
Large-scale 3D semantic scene generation has predominantly relied on voxel-based representations, which are memory-intensive, bound by fixed resolutions, and challenging to edit. In contrast, primitives represent semantic entities using compact, coarse 3D structures that are easy to manipulate and compose, making them an ideal representation for this task. In this paper, we introduce PrITTI, a latent diffusion-based framework that leverages primitives as the main foundational elements for generating compositional, controllable, and editable 3D semantic scene layouts. Our method adopts a hybrid representation, modeling ground surfaces in a rasterized format while encoding objects as vectorized 3D primitives. This decomposition is also reflected in a structured latent representation that enables flexible scene manipulation of ground and object components. To overcome the orientation ambiguities in conventional encoding methods, we introduce a stable Cholesky-based parameterization that jointly encodes object size and orientation. Experiments on the KITTI-360 dataset show that PrITTI outperforms a voxel-based baseline in generation quality, while reducing memory requirements by up to $3\times$. In addition, PrITTI enables direct instance-level manipulation of objects in the scene and supports a range of downstream applications, including scene inpainting, outpainting, and photo-realistic street-view synthesis.
Abstract (translated)
大规模的三维语义场景生成主要依赖于基于体素（voxel）的表示方法，这种方法内存消耗大、分辨率固定且难以编辑。相比之下，原语（primitives）使用紧凑而粗糙的3D结构来表示语义实体，易于操作和组合，成为这一任务的理想选择。在本文中，我们介绍了一种名为PrITTI的新框架，该框架采用基于潜在扩散的方法，并以原语作为生成组成化、可控性和可编辑性三维语义场景布局的主要基础元素。我们的方法采用了混合表示方式，在栅格格式下建模地面表面的同时将物体编码为向量化的3D原语。这种分解也反映在一种结构化的潜在表示中，该表示能够灵活地操作场景中的地面和对象组件。 为了克服传统编码方法中存在的方位模糊性问题，我们引入了一种基于稳定的乔累斯基（Cholesky）参数化的方法，这种方法同时编码物体的大小和方向。实验结果表明，在KITTI-360数据集上的生成质量方面，PrITTI优于基于体素的基础线模型，并且将内存需求减少了最多三倍。此外，PrITTI支持场景中对象实例级别的直接操作，并适用于包括场景修复、扩展以及合成照片级真实街道视图在内的多种下游应用。
URL
https://arxiv.org/abs/2506.19117
