Abstract
Text-to-3D asset generation has achieved significant optimization under the supervision of 2D diffusion priors. However, when dealing with compositional scenes, existing methods encounter several challenges: 1). failure to ensure that composite scene layouts comply with physical laws; 2). difficulty in accurately capturing the assets and relationships described in complex scene descriptions; 3). limited autonomous asset generation capabilities among layout approaches leveraging large language models (LLMs). To avoid these compromises, we propose a novel framework for compositional scene generation, PhiP-G, which seamlessly integrates generation techniques with layout guidance based on a world model. Leveraging LLM-based agents, PhiP-G analyzes the complex scene description to generate a scene graph, and integrating a multimodal 2D generation agent and a 3D Gaussian generation method for targeted assets creation. For the stage of layout, PhiP-G employs a physical pool with adhesion capabilities and a visual supervision agent, forming a world model for layout prediction and planning. Extensive experiments demonstrate that PhiP-G significantly enhances the generation quality and physical rationality of the compositional scenes. Notably, PhiP-G attains state-of-the-art (SOTA) performance in CLIP scores, achieves parity with the leading methods in generation quality as measured by the T$^3$Bench, and improves efficiency by 24x.
Abstract (translated)
在二维扩散先验的监督下，文本到三维资产生成已经实现了显著优化。然而，在处理组合场景时，现有方法面临以下挑战：1) 无法确保复合场景布局符合物理规律；2) 难以准确捕捉复杂场景描述中的资产和关系；3) 利用大型语言模型（LLMs）的布局方法在自主资产生成能力方面存在局限。为避免这些妥协，我们提出了一种新颖的框架PhiP-G，该框架无缝地将生成技术与基于世界模型的布局指导相结合。 通过利用基于大语言模型的代理，PhiP-G 分析复杂场景描述来生成场景图，并整合多模态二维生成代理和三维高斯生成方法以创建特定资产。在布局阶段，PhiP-G 采用具有粘合能力的物理池和视觉监督代理，形成一个世界模型用于布局预测和规划。 广泛的实验表明，PhiP-G 显著提升了组合场景生成的质量和物理合理性。特别地，PhiP-G 在CLIP评分中达到最先进的（SOTA）性能，在T$^3$Bench 测量的生成质量上与领先方法持平，并且在效率方面提高了24倍。
URL
https://arxiv.org/abs/2502.00708
