Abstract
Large-scale scene data is essential for training and testing in robot learning. Neural reconstruction methods have promised the capability of reconstructing large physically-grounded outdoor scenes from captured sensor data. However, these methods have baked-in static environments and only allow for limited scene control -- they are functionally constrained in scene and trajectory diversity by the captures from which they are reconstructed. In contrast, generating driving data with recent image or video diffusion models offers control, however, at the cost of geometry grounding and causality. In this work, we aim to bridge this gap and present a method that directly generates large-scale 3D driving scenes with accurate geometry, allowing for causal novel view synthesis with object permanence and explicit 3D geometry estimation. The proposed method combines the generation of a proxy geometry and environment representation with score distillation from learned 2D image priors. We find that this approach allows for high controllability, enabling the prompt-guided geometry and high-fidelity texture and structure that can be conditioned on map layouts -- producing realistic and geometrically consistent 3D generations of complex driving scenes.
Abstract (translated)
大规模场景数据对于机器人学习的训练和测试至关重要。神经重建方法承诺可以从捕获的传感器数据中重构大型且物理基础坚实的室外场景。然而，这些方法内置了静态环境限制，并仅允许有限的场景控制——它们的功能受限于用于重建的数据捕捉中的多样性和轨迹多样性。 相比之下，使用最近的图像或视频扩散模型生成驾驶数据可以提供更多的控制能力，但代价是失去了几何精确度和因果关系。在这项工作中，我们旨在弥合这一差距，并提出了一种直接生成大规模3D驾驶场景的方法，该方法具有准确的几何形状，并且能够进行因果驱动的新视图合成、物体持久性和显式的3D几何估计。 所提出的这种方法结合了代理几何体和环境表示的生成以及从学习到的2D图像先验中提取分数。我们发现，这种方法允许高度可控性，可以基于地图布局对提示引导下的几何形状和高保真的纹理及结构进行条件设定——从而能够产生现实且几何上一致的大规模驾驶场景的3D生成。
URL
https://arxiv.org/abs/2508.19204
