Abstract
We present Free4D, a novel tuning-free framework for 4D scene generation from a single image. Existing methods either focus on object-level generation, making scene-level generation infeasible, or rely on large-scale multi-view video datasets for expensive training, with limited generalization ability due to the scarcity of 4D scene data. In contrast, our key insight is to distill pre-trained foundation models for consistent 4D scene representation, which offers promising advantages such as efficiency and generalizability. 1) To achieve this, we first animate the input image using image-to-video diffusion models followed by 4D geometric structure initialization. 2) To turn this coarse structure into spatial-temporal consistent multiview videos, we design an adaptive guidance mechanism with a point-guided denoising strategy for spatial consistency and a novel latent replacement strategy for temporal coherence. 3) To lift these generated observations into consistent 4D representation, we propose a modulation-based refinement to mitigate inconsistencies while fully leveraging the generated information. The resulting 4D representation enables real-time, controllable rendering, marking a significant advancement in single-image-based 4D scene generation.
Abstract (translated)
我们提出了一种名为Free4D的新框架，该框架无需调优即可从单张图像生成4D场景。现有方法要么专注于对象级别的生成，使得场景级别的生成变得不可行；要么依赖于大规模的多视角视频数据集进行昂贵的训练，并且由于缺乏4D场景数据而具有有限的泛化能力。相比之下，我们的关键洞察是通过蒸馏预训练的基础模型来实现一致的4D场景表示，这种方法提供了诸如效率和泛化能力等显著优势。 为了实现这一目标： 1. 我们首先使用图像到视频的扩散模型对输入图像进行动画处理，并随后采用4D几何结构初始化。 2. 为将这种粗略结构转变为时空一致的多视角视频，我们设计了一种自适应引导机制，包括用于空间一致性的一点引导去噪策略和一种新颖的潜在替换策略以实现时间连贯性。 3. 要将这些生成的观察提升到一致的4D表示中，我们提出了一种基于调制的细化方法，通过这种方式可以缓解不一致性，并充分利用生成的信息。 最终得到的4D表示能够实现实时、可控渲染，在单张图像基础上进行4D场景生成方面取得了重大进展。
URL
https://arxiv.org/abs/2503.20785
