Abstract
In this paper, we introduce \textbf{DimensionX}, a framework designed to generate photorealistic 3D and 4D scenes from just a single image with video diffusion. Our approach begins with the insight that both the spatial structure of a 3D scene and the temporal evolution of a 4D scene can be effectively represented through sequences of video frames. While recent video diffusion models have shown remarkable success in producing vivid visuals, they face limitations in directly recovering 3D/4D scenes due to limited spatial and temporal controllability during generation. To overcome this, we propose ST-Director, which decouples spatial and temporal factors in video diffusion by learning dimension-aware LoRAs from dimension-variant data. This controllable video diffusion approach enables precise manipulation of spatial structure and temporal dynamics, allowing us to reconstruct both 3D and 4D representations from sequential frames with the combination of spatial and temporal dimensions. Additionally, to bridge the gap between generated videos and real-world scenes, we introduce a trajectory-aware mechanism for 3D generation and an identity-preserving denoising strategy for 4D generation. Extensive experiments on various real-world and synthetic datasets demonstrate that DimensionX achieves superior results in controllable video generation, as well as in 3D and 4D scene generation, compared with previous methods.
Abstract (translated)
在这篇论文中，我们介绍了**DimensionX**框架，该框架旨在通过仅使用带有视频扩散的单个图像来生成逼真的3D和4D场景。我们的方法基于这样的见解：即一个3D场景的空间结构以及一个4D场景的时间演变可以通过一系列视频帧有效地表示出来。尽管近期的视频扩散模型在产生生动的视觉效果方面取得了显著成功，但由于在生成过程中存在空间和时间控制方面的限制，它们直接恢复3D/4D场景的能力受到了限制。为克服这一问题，我们提出了ST-Director，它通过从维度变化的数据中学习维度感知LoRAs（低秩适配器），从而将视频扩散中的空间因素与时间因素解耦。这种可控的视频扩散方法能够精确操控空间结构和时间动态，使我们能结合空间和时间维度从连续帧重建3D和4D表示。此外，为了弥合生成视频与现实场景之间的差距，我们引入了针对3D生成的轨迹感知机制以及用于4D生成的身份保持去噪策略。在各种真实世界和合成数据集上的广泛实验表明，DimensionX相较于之前的方法，在可控视频生成以及3D和4D场景生成方面取得了更优的结果。
URL
https://arxiv.org/abs/2411.04928
