Abstract
In this work, we introduce Prometheus, a 3D-aware latent diffusion model for text-to-3D generation at both object and scene levels in seconds. We formulate 3D scene generation as multi-view, feed-forward, pixel-aligned 3D Gaussian generation within the latent diffusion paradigm. To ensure generalizability, we build our model upon pre-trained text-to-image generation model with only minimal adjustments, and further train it using a large number of images from both single-view and multi-view datasets. Furthermore, we introduce an RGB-D latent space into 3D Gaussian generation to disentangle appearance and geometry information, enabling efficient feed-forward generation of 3D Gaussians with better fidelity and geometry. Extensive experimental results demonstrate the effectiveness of our method in both feed-forward 3D Gaussian reconstruction and text-to-3D generation. Project page: this https URL
Abstract (translated)
在这项工作中，我们介绍了Prometheus，这是一种具有三维感知的潜在扩散模型，可在几秒钟内生成从物体到场景级别的文本到3D内容。我们将3D场景生成视为在潜在扩散框架内的多视角、前馈、像素对齐的3D高斯生成。为了确保泛化能力，我们在预训练的文本到图像生成模型的基础上构建了我们的模型，并仅进行了少量调整，之后使用来自单视图和多视图数据集的大规模图像对其进行进一步训练。此外，我们还将RGB-D潜在空间引入到3D高斯生成中，以分离外观信息与几何信息，从而能够高效地前馈生成具有更高保真度和几何结构的3D高斯模型。广泛的实验结果证明了我们的方法在前馈3D高斯重建及文本到3D生成方面的有效性。项目页面：[此链接](https://this-url.com/)
URL
https://arxiv.org/abs/2412.21117
