Abstract
Synthesizing photo-realistic visual observations from an ego vehicle's driving trajectory is a critical step towards scalable training of self-driving models. Reconstruction-based methods create 3D scenes from driving logs and synthesize geometry-consistent driving videos through neural rendering, but their dependence on costly object annotations limits their ability to generalize to in-the-wild driving scenarios. On the other hand, generative models can synthesize action-conditioned driving videos in a more generalizable way but often struggle with maintaining 3D visual consistency. In this paper, we present DreamDrive, a 4D spatial-temporal scene generation approach that combines the merits of generation and reconstruction, to synthesize generalizable 4D driving scenes and dynamic driving videos with 3D consistency. Specifically, we leverage the generative power of video diffusion models to synthesize a sequence of visual references and further elevate them to 4D with a novel hybrid Gaussian representation. Given a driving trajectory, we then render 3D-consistent driving videos via Gaussian splatting. The use of generative priors allows our method to produce high-quality 4D scenes from in-the-wild driving data, while neural rendering ensures 3D-consistent video generation from the 4D scenes. Extensive experiments on nuScenes and street view images demonstrate that DreamDrive can generate controllable and generalizable 4D driving scenes, synthesize novel views of driving videos with high fidelity and 3D consistency, decompose static and dynamic elements in a self-supervised manner, and enhance perception and planning tasks for autonomous driving.
Abstract (translated)
从自动驾驶车辆的驾驶轨迹中合成逼真的视觉观测结果是可扩展训练自动驾驶模型的关键步骤。基于重建的方法通过神经渲染技术，可以从行驶日志创建3D场景并生成几何一致性的行驶视频，但它们依赖于昂贵的对象标注限制了其在现实环境中的一般化能力。另一方面，生成式模型可以通过一种更为通用的方式合成受动作条件约束的行驶视频，但在保持三维视觉一致性方面往往遇到困难。 在这篇论文中，我们提出了DreamDrive这一4D时空场景生成方法，它结合了生成和重建的优点，从而可以综合地产生具有3D一致性的通用化4D驾驶场景以及动态驾驶视频。具体而言，我们利用视频扩散模型的生成能力来合成一系列视觉参考，并通过一种新颖的混合高斯表示法将其提升为4D。在给定驾驶轨迹之后，我们进一步使用高斯点光源技术渲染出三维一致性良好的驾驶视频。这种方法借助于生成先验知识可以从现实中的行驶数据中产生高质量的4D场景，而神经渲染则确保从这些4D场景生成具有3D一致性的视频。 大量的实验表明，在nuScenes和街景图像上，DreamDrive可以生成可控且通用化的4D驾驶场景，以高保真度及3D一致性合成新的行驶视图，并在自监督模式下分解静态与动态元素。此外，它还能增强自动驾驶中的感知和规划任务。
URL
https://arxiv.org/abs/2501.00601
