Abstract
Recent advancements in 3D object generation using diffusion models have achieved remarkable success, but generating realistic 3D urban scenes remains challenging. Existing methods relying solely on 3D diffusion models tend to suffer a degradation in appearance details, while those utilizing only 2D diffusion models typically compromise camera controllability. To overcome this limitation, we propose ScenDi, a method for urban scene generation that integrates both 3D and 2D diffusion models. We first train a 3D latent diffusion model to generate 3D Gaussians, enabling the rendering of images at a relatively low resolution. To enable controllable synthesis, this 3DGS generation process can be optionally conditioned by specifying inputs such as 3d bounding boxes, road maps, or text prompts. Then, we train a 2D video diffusion model to enhance appearance details conditioned on rendered images from the 3D Gaussians. By leveraging the coarse 3D scene as guidance for 2D video diffusion, ScenDi generates desired scenes based on input conditions and successfully adheres to accurate camera trajectories. Experiments on two challenging real-world datasets, Waymo and KITTI-360, demonstrate the effectiveness of our approach.
Abstract (translated)
最近使用扩散模型生成三维物体取得了显著的成功，但生成逼真的三维城市场景依然具有挑战性。现有的仅依赖于3D扩散模型的方法往往会损失外观细节的清晰度，而那些仅仅利用2D扩散模型的方法则会牺牲相机控制能力。为了克服这一局限性，我们提出了ScenDi方法，这是一种结合了3D和2D扩散模型的城市场景生成技术。 首先，我们训练一个三维潜在扩散模型来生成三维高斯分布，这使得以相对较低的分辨率渲染图像成为可能。为了实现可控合成，这个三维高斯生成过程可以被指定输入（如三维边界框、道路地图或文本提示）所条件化。接着，我们训练了一个2D视频扩散模型，在此基础上提升从3D高斯渲染出的图像中的外观细节。 通过利用粗略的3D场景作为引导来指导2D视频扩散，ScenDi能够根据输入条件生成所需的场景，并且成功地遵循精确的相机轨迹。我们在两个具有挑战性的真实数据集（Waymo和KITTI-360）上的实验结果证明了我们方法的有效性。
URL
https://arxiv.org/abs/2601.15221
