Abstract
Recent diffusion models have demonstrated remarkable performance in both 3D scene generation and perception tasks. Nevertheless, existing methods typically separate these two processes, acting as a data augmenter to generate synthetic data for downstream perception tasks. In this work, we propose OccScene, a novel mutual learning paradigm that integrates fine-grained 3D perception and high-quality generation in a unified framework, achieving a cross-task win-win effect. OccScene generates new and consistent 3D realistic scenes only depending on text prompts, guided with semantic occupancy in a joint-training diffusion framework. To align the occupancy with the diffusion latent, a Mamba-based Dual Alignment module is introduced to incorporate fine-grained semantics and geometry as perception priors. Within OccScene, the perception module can be effectively improved with customized and diverse generated scenes, while the perception priors in return enhance the generation performance for mutual benefits. Extensive experiments show that OccScene achieves realistic 3D scene generation in broad indoor and outdoor scenarios, while concurrently boosting the perception models to achieve substantial performance improvements in the 3D perception task of semantic occupancy prediction.
Abstract (translated)
最近的扩散模型在3D场景生成和感知任务中展现了出色的性能。然而，现有方法通常将这两个过程分开，作为数据增强器来为下游感知任务生成合成数据。在这项工作中，我们提出了OccScene，一个新颖的相互学习范式，在统一框架内整合了细粒度的3D感知和高质量生成，实现了跨任务双赢效果。OccScene仅依赖文本提示即可生成新的、一致的3D现实场景，并通过语义占用在联合训练扩散框架中进行指导。为了将占用与扩散潜在空间对齐，我们引入了一个基于Mamba的双重对齐模块，以结合细粒度的语义和几何作为感知先验。在OccScene内部，感知模块可以通过定制化和多样化的生成场景得到有效提升，而反过来，这些感知先验又会增强生成性能，实现相互受益。大量的实验表明，OccScene能够实现在广泛室内和室外场景下的现实3D场景生成，并且同时提升了感知模型，在语义占用预测的3D感知任务中实现了显著的性能改进。
URL
https://arxiv.org/abs/2412.11183
