Abstract
We introduce RealmDreamer, a technique for generation of general forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D Gaussian Splatting representation to match complex text prompts. We initialize these splats by utilizing the state-of-the-art text-to-image generators, lifting their samples into 3D, and computing the occlusion volume. We then optimize this representation across multiple views as a 3D inpainting task with image-conditional diffusion models. To learn correct geometric structure, we incorporate a depth diffusion model by conditioning on the samples from the inpainting model, giving rich geometric structure. Finally, we finetune the model using sharpened samples from image generators. Notably, our technique does not require video or multi-view data and can synthesize a variety of high-quality 3D scenes in different styles, consisting of multiple objects. Its generality additionally allows 3D synthesis from a single image.
Abstract (translated)
我们介绍了一种从文本描述生成通用面向未来的3D场景的技术，名为RealmDreamer。我们的技术通过优化3D高斯平铺表示来匹配复杂的文本提示。我们通过利用最先进的文本到图像生成器的状态，将样本提升到3D并计算遮挡体积。然后，在多个视角上对这种表示进行优化，将其作为3D修复任务与图像条件扩散模型一起进行。为了学习正确的几何结构，我们在修复模型上通过条件于修复模型的样本，从而赋予了丰富几何结构。最后，我们使用增强的生成器样本对模型进行微调。值得注意的是，我们的技术不需要视频或多视角数据，可以生成不同风格的高质量3D场景，包括多个物体。此外，其普遍性还允许从单张图像合成3D。
URL
https://arxiv.org/abs/2404.07199
