Abstract
Recently, the field of text-guided 3D scene generation has garnered significant attention. High-quality generation that aligns with physical realism and high controllability is crucial for practical 3D scene applications. However, existing methods face fundamental limitations: (i) difficulty capturing complex relationships between multiple objects described in the text, (ii) inability to generate physically plausible scene layouts, and (iii) lack of controllability and extensibility in compositional scenes. In this paper, we introduce LayoutDreamer, a framework that leverages 3D Gaussian Splatting (3DGS) to facilitate high-quality, physically consistent compositional scene generation guided by text. Specifically, given a text prompt, we convert it into a directed scene graph and adaptively adjust the density and layout of the initial compositional 3D Gaussians. Subsequently, dynamic camera adjustments are made based on the training focal point to ensure entity-level generation quality. Finally, by extracting directed dependencies from the scene graph, we tailor physical and layout energy to ensure both realism and flexibility. Comprehensive experiments demonstrate that LayoutDreamer outperforms other compositional scene generation quality and semantic alignment methods. Specifically, it achieves state-of-the-art (SOTA) performance in the multiple objects generation metric of T3Bench.
Abstract (translated)
最近，文本引导的三维场景生成领域引起了广泛关注。高质量地生成符合物理真实性和高度可控性的三维场景对于实际应用至关重要。然而，现有的方法面临一些根本性的限制：（i）难以捕捉文本描述中多个对象之间的复杂关系；（ii）无法生成合乎物理逻辑的场景布局；（iii）在组合式场景中的可控制性与扩展性不足。为此，在本文中我们介绍了LayoutDreamer框架，该框架利用3D高斯点阵(3DGS)技术来实现高质量、物理上一致且由文本引导的组合式三维场景生成。具体而言，给定一个文本提示后，我们将它转换为有向场景图，并根据初始的组合式3D高斯分布自适应地调整其密度和布局。随后，基于训练焦点进行动态相机调整以确保实体级别的生成质量。最后，通过从场景图中提取定向依赖关系，我们定制物理能量与布局能量，以此保证现实性和灵活性。全面的实验表明，LayoutDreamer在组合式三维场景生成质量和语义对齐方法方面优于其他模型，并且在T3Bench的多对象生成度量标准上达到了最新的技术水平（SOTA）。
URL
https://arxiv.org/abs/2502.01949
