Abstract
Controllable 3D scene generation has extensive applications in virtual reality and interior design, where the generated scenes should exhibit high levels of realism and controllability in terms of geometry. Scene graphs provide a suitable data representation that facilitates these applications. However, current graph-based methods for scene generation are constrained to text-based inputs and exhibit insufficient adaptability to flexible user inputs, hindering the ability to precisely control object geometry. To address this issue, we propose MMGDreamer, a dual-branch diffusion model for scene generation that incorporates a novel Mixed-Modality Graph, visual enhancement module, and relation predictor. The mixed-modality graph allows object nodes to integrate textual and visual modalities, with optional relationships between nodes. It enhances adaptability to flexible user inputs and enables meticulous control over the geometry of objects in the generated scenes. The visual enhancement module enriches the visual fidelity of text-only nodes by constructing visual representations using text embeddings. Furthermore, our relation predictor leverages node representations to infer absent relationships between nodes, resulting in more coherent scene layouts. Extensive experimental results demonstrate that MMGDreamer exhibits superior control of object geometry, achieving state-of-the-art scene generation performance. Project page: this https URL.
Abstract (translated)
可控的3D场景生成在虚拟现实和室内设计等领域有着广泛的应用，生成的场景需要表现出高度的真实感与几何形状上的可控制性。场景图提供了一种合适的数据表示方式，便于实现这些应用需求。然而，当前基于图形的方法进行场景生成时，受限于文本输入，并且对于灵活多变的用户输入适应性不足，这限制了精确控制物体几何结构的能力。 为了解决这个问题，我们提出了MMGDreamer，这是一种用于场景生成的双分支扩散模型，集成了新颖的混合模态图、视觉增强模块和关系预测器。这种混合模态图允许对象节点整合文本和视觉两种模式，并支持节点间的可选关系连接。这提升了对灵活用户输入的适应性，并使得精确控制生成场景中物体的几何形状成为可能。 此外，MMGDreamer中的视觉增强模块通过使用文本嵌入来构建视觉表示，从而丰富了仅基于文本节点的视觉保真度。我们的关系预测器利用节点表示来推断节点间缺失的关系，进而产生更为连贯的场景布局。 广泛的实验结果表明，MMGDreamer在物体几何控制方面表现出色，并达到了目前最先进的场景生成性能水平。项目页面链接：[此URL](this https URL)（请替换为实际链接）。
URL
https://arxiv.org/abs/2502.05874
