Abstract
Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization. A key challenge lies in finding an efficient and generalizable geometric representation that seamlessly connects temporal and spatial synthesis. To address this, we propose DiST-4D, the first disentangled spatiotemporal diffusion framework for 4D driving scene generation, which leverages metric depth as the core geometric representation. DiST-4D decomposes the problem into two diffusion processes: DiST-T, which predicts future metric depth and multi-view RGB sequences directly from past observations, and DiST-S, which enables spatial NVS by training only on existing viewpoints while enforcing cycle consistency. This cycle consistency mechanism introduces a forward-backward rendering constraint, reducing the generalization gap between observed and unseen viewpoints. Metric depth is essential for both accurate reliable forecasting and accurate spatial NVS, as it provides a view-consistent geometric representation that generalizes well to unseen perspectives. Experiments demonstrate that DiST-4D achieves state-of-the-art performance in both temporal prediction and NVS tasks, while also delivering competitive performance in planning-related evaluations.
Abstract (translated)
当前的生成模型在合成动态4D驾驶场景时遇到了挑战，这些场景需要同时支持时间外推和空间新视角合成（NVS），而无需针对每个场景进行优化。关键难题在于寻找一种高效且通用化的几何表示方法，能够无缝地连接时间与空间的合成过程。为了解决这一问题，我们提出了DiST-4D，这是一种用于生成4D驾驶场景的分解时空扩散框架，它利用了度量深度作为核心几何表示。 DiST-4D将问题分解成两个扩散过程：DiST-T和DiST-S。其中，DiST-T能够直接从过去的观测数据预测未来的度量深度以及多视角RGB序列；而DiST-S则通过仅在现有视角上训练并强制执行循环一致性来实现空间NVS。这种循环一致性机制引入了前向-后向渲染约束，从而减少了观察到的和未见过的观点之间的泛化差距。 度量深度对于准确可靠的预测和精确的空间NVS都是至关重要的，因为它提供了一种视图一致性的几何表示方法，并且能够很好地推广应用于未知视角。实验结果表明，DiST-4D在时间预测任务和NVS任务中都达到了最先进的性能水平，在与规划相关的评估方面也表现出竞争性表现。
URL
https://arxiv.org/abs/2503.15208
