Abstract
Most 3D scene generation methods are limited to only generating object bounding box parameters while newer diffusion methods also generate class labels and latent features. Using object size or latent feature, they then retrieve objects from a predefined database. For complex scenes of varied, multi-categorical objects, diffusion-based latents cannot be effectively decoded by current autoencoders into the correct point cloud objects which agree with target classes. We introduce a Class-Partitioned Vector Quantized Variational Autoencoder (CPVQ-VAE) that is trained to effectively decode object latent features, by employing a pioneering $\textit{class-partitioned codebook}$ where codevectors are labeled by class. To address the problem of $\textit{codebook collapse}$, we propose a $\textit{class-aware}$ running average update which reinitializes dead codevectors within each partition. During inference, object features and class labels, both generated by a Latent-space Flow Matching Model (LFMM) designed specifically for scene generation, are consumed by the CPVQ-VAE. The CPVQ-VAE's class-aware inverse look-up then maps generated latents to codebook entries that are decoded to class-specific point cloud shapes. Thereby, we achieve pure point cloud generation without relying on an external objects database for retrieval. Extensive experiments reveal that our method reliably recovers plausible point cloud scenes, with up to 70.4% and 72.3% reduction in Chamfer and Point2Mesh errors on complex living room scenes.
Abstract (translated)
大多数的三维场景生成方法仅限于生成物体边界框参数，而较新的扩散方法还能生成类别标签和潜在特征。利用这些对象大小或潜在特征信息，它们从预定义数据库中检索对象。对于包含多样且多类别的复杂场景而言，基于扩散的方法无法通过当前自动编码器有效解码为符合目标类别的点云物体。 我们引入了一种名为分类分区向量量化变分自编码器（Class-Partitioned Vector Quantized Variational Autoencoder, CPVQ-VAE）的模型。它通过使用一种创新性的按类别分割代码簿，其中每个代码向量都由其所属类别标记来进行训练，从而能够有效地解码对象潜在特征。 为了解决“代码簿塌陷”问题，我们提出了一种“类别感知”的运行平均更新方法，这种方法可以重新初始化每个分区内的无效代码向量。在推理阶段，CPVQ-VAE会接收由专门用于场景生成的潜在空间流匹配模型（Latent-space Flow Matching Model, LFMM）产生的对象特征和类别标签。 通过使用具有类别感知逆查找功能的CPVQ-VAE，所生成的潜在特性会被映射到代码簿中的相应条目，并且这些条目随后被解码为特定类别的点云形状。这样一来，我们就能实现纯粹基于点云的场景生成，而无需依赖外部对象数据库。 大量的实验表明，我们的方法能够可靠地恢复出合理的点云场景，在复杂的客厅环境中分别减少了高达70.4%和72.3%的Chamfer误差和Point2Mesh误差。
URL
https://arxiv.org/abs/2601.12391
