Abstract
Existing 4D synthesis methods primarily focus on object-level generation or dynamic scene synthesis with limited novel views, restricting their ability to generate multi-view consistent and immersive dynamic 4D scenes. To address these constraints, we propose a framework (dubbed as CoCo4D) for generating detailed dynamic 4D scenes from text prompts, with the option to include images. Our method leverages the crucial observation that articulated motion typically characterizes foreground objects, whereas background alterations are less pronounced. Consequently, CoCo4D divides 4D scene synthesis into two responsibilities: modeling the dynamic foreground and creating the evolving background, both directed by a reference motion sequence. Given a text prompt and an optional reference image, CoCo4D first generates an initial motion sequence utilizing video diffusion models. This motion sequence then guides the synthesis of both the dynamic foreground object and the background using a novel progressive outpainting scheme. To ensure seamless integration of the moving foreground object within the dynamic background, CoCo4D optimizes a parametric trajectory for the foreground, resulting in realistic and coherent blending. Extensive experiments show that CoCo4D achieves comparable or superior performance in 4D scene generation compared to existing methods, demonstrating its effectiveness and efficiency. More results are presented on our website this https URL.
Abstract (translated)
现有的4D合成方法主要集中在对象级别的生成或具有有限新颖视角的动态场景合成上，这限制了它们生成多视角一致和沉浸式的动态4D场景的能力。为了解决这些局限性，我们提出了一种框架（称为CoCo4D），该框架可以根据文本提示生成详细的动态4D场景，并可选择性地包含图像。我们的方法基于一个关键观察：连贯运动通常表征前景对象的特性，而背景的变化则相对较少。因此，CoCo4D将4D场景合成分为两个任务：建模动态前景和创造不断变化的背景，两者都由参考运动序列指导。 给定文本提示及可选参考图像后，CoCo4D首先使用视频扩散模型生成初始运动序列。接着，该运动序列指导动态前景对象和背景的合成过程，采用了一种新颖的渐进式扩展方案（progressive outpainting scheme）。为了确保移动前景物体在动态背景下无缝融合，CoCo4D优化了前景路径的参数化轨迹，从而实现了现实感强且连贯的合成效果。 广泛的实验表明，与现有方法相比，CoCo4D在生成4D场景方面取得了相当或更好的性能表现，证明了其有效性和效率。更多结果可在我们的网站上查看：[此URL]（请将"this https URL"替换为实际链接）。
URL
https://arxiv.org/abs/2506.19798
