Abstract
Recent advancements in diffusion models for 2D and 3D content creation have sparked a surge of interest in generating 4D content. However, the scarcity of 3D scene datasets constrains current methodologies to primarily object-centric generation. To overcome this limitation, we present Comp4D, a novel framework for Compositional 4D Generation. Unlike conventional methods that generate a singular 4D representation of the entire scene, Comp4D innovatively constructs each 4D object within the scene separately. Utilizing Large Language Models (LLMs), the framework begins by decomposing an input text prompt into distinct entities and maps out their trajectories. It then constructs the compositional 4D scene by accurately positioning these objects along their designated paths. To refine the scene, our method employs a compositional score distillation technique guided by the pre-defined trajectories, utilizing pre-trained diffusion models across text-to-image, text-to-video, and text-to-3D domains. Extensive experiments demonstrate our outstanding 4D content creation capability compared to prior arts, showcasing superior visual quality, motion fidelity, and enhanced object interactions.
Abstract (translated)
近年来，在二维和三维内容创作中扩散模型的最新进展引发了人们对生成4D内容的浓厚兴趣。然而，3D场景数据集的稀缺性限制了当前方法主要集中在以物体为中心的生成。为了克服这一限制，我们提出了Comp4D，一种用于合成4D内容的全新框架。与传统方法生成整个场景的单一4D表示不同，Comp4D创新地构建了场景中的每个4D物体。利用大型语言模型（LLMs），该框架首先将输入文本提示分解为不同的实体，并对其轨迹进行探索。然后，它准确地将这些物体沿着其预设路径定位，构建了合成4D场景。为了优化场景，我们的方法采用了一个基于预定义轨迹的合成分数蒸馏技术，利用了跨文本到图像、文本到视频和文本到3D领域的预训练扩散模型。大量实验证明，与以前的艺术作品相比，我们的4D内容创作能力非凡，展示了卓越的视觉质量、动作流畅性和增强的物体交互。
URL
https://arxiv.org/abs/2403.16993
