Abstract
Three-dimensional scene generation is crucial in computer vision, with applications spanning autonomous driving, gaming and the metaverse. Current methods either lack user control or rely on imprecise, non-intuitive conditions. In this work, we propose a method that uses, scene graphs, an accessible, user friendly control format to generate outdoor 3D scenes. We develop an interactive system that transforms a sparse scene graph into a dense BEV (Bird's Eye View) Embedding Map, which guides a conditional diffusion model to generate 3D scenes that match the scene graph description. During inference, users can easily create or modify scene graphs to generate large-scale outdoor scenes. We create a large-scale dataset with paired scene graphs and 3D semantic scenes to train the BEV embedding and diffusion models. Experimental results show that our approach consistently produces high-quality 3D urban scenes closely aligned with the input scene graphs. To the best of our knowledge, this is the first approach to generate 3D outdoor scenes conditioned on scene graphs.
Abstract (translated)
三维场景生成在计算机视觉领域至关重要，其应用范围涵盖自动驾驶、游戏以及元宇宙。当前的方法要么缺乏用户控制能力，要么依赖于不精确且不易理解的条件。在这项工作中，我们提出了一种使用场景图（一种易用且友好的用户控制格式）来生成户外3D场景的方法。我们开发了一个交互式系统，该系统将稀疏的场景图转换为密集的BEV（鸟瞰视图）嵌入地图，并指导有条件的扩散模型生成与场景图描述相匹配的3D场景。在推理过程中，用户可以轻松创建或修改场景图以生成大规模户外场景。我们创建了一个大型数据集，其中包括配对的场景图和三维语义场景，用于训练BEV嵌入和扩散模型。实验结果表明，我们的方法始终能够产生高质量、与输入场景图紧密一致的3D城市场景。据我们所知，这是第一个基于场景图生成三维户外场景的方法。
URL
https://arxiv.org/abs/2503.07152
