Abstract
Recent advancements in object-centric text-to-3D generation have shown impressive results. However, generating complex 3D scenes remains an open challenge due to the intricate relations between objects. Moreover, existing methods are largely based on score distillation sampling (SDS), which constrains the ability to manipulate multiobjects with specific interactions. Addressing these critical yet underexplored issues, we present a novel framework of Scene Graph and Layout Guided 3D Scene Generation (GraLa3D). Given a text prompt describing a complex 3D scene, GraLa3D utilizes LLM to model the scene using a scene graph representation with layout bounding box information. GraLa3D uniquely constructs the scene graph with single-object nodes and composite super-nodes. In addition to constraining 3D generation within the desirable layout, a major contribution lies in the modeling of interactions between objects in a super-node, while alleviating appearance leakage across objects within such nodes. Our experiments confirm that GraLa3D overcomes the above limitations and generates complex 3D scenes closely aligned with text prompts.
Abstract (translated)
最近在基于对象的文本到3D生成领域的进展已经取得了令人印象深刻的结果。然而，由于物体之间的复杂关系，生成复杂的3D场景仍然是一项挑战。此外，现有的方法大多基于得分蒸馏采样（SDS），这限制了多物体间特定交互操作的能力。为了应对这些关键但尚未充分探索的问题，我们提出了一种新的框架——场景图和布局引导的3D场景生成（GraLa3D）。给定一个描述复杂3D场景的文本提示，GraLa3D利用大规模语言模型（LLM）通过带有布局边界框信息的场景图来建模场景。GraLa3D独特地构建了具有单个对象节点和复合超节点的场景图。除了在期望的布局内约束3D生成之外，主要贡献在于能够建模超节点中物体之间的交互，并缓解这些节点内部各物体之间出现泄漏的问题。实验结果证实，GraLa3D克服了上述限制，并能够生成与文本提示紧密相关的复杂3D场景。
URL
https://arxiv.org/abs/2412.20473
