Abstract
Recent progress in image and video synthesis has inspired their use in advancing 3D scene generation. However, we observe that text-to-image and -video approaches struggle to maintain scene- and object-level consistency beyond a limited environment scale due to the absence of explicit geometry. We thus present a geometry-first approach that decouples this complex problem of large-scale 3D scene synthesis into its structural composition, represented as a mesh scaffold, and realistic appearance synthesis, which leverages powerful image synthesis models conditioned on the mesh scaffold. From an input text description, we first construct a mesh capturing the environment's geometry (walls, floors, etc.), and then use image synthesis, segmentation and object reconstruction to populate the mesh structure with objects in realistic layouts. This mesh scaffold is then rendered to condition image synthesis, providing a structural backbone for consistent appearance generation. This enables scalable, arbitrarily-sized 3D scenes of high object richness and diversity, combining robust 3D consistency with photorealistic detail. We believe this marks a significant step toward generating truly environment-scale, immersive 3D worlds.
Abstract (translated)
图像与视频合成的最新进展推动了其在3D场景生成领域的应用。然而，我们观察到由于缺乏显式几何信息，现有的文生图与文生视频方法在环境规模超出一定范围后，难以保持场景级与对象级的一致性。因此，我们提出一种几何优先的方法，将大规模3D场景合成这一复杂问题解耦为结构组成（以网格支架表示）与真实外观合成（利用基于网格支架条件的强大图像合成模型）。从输入的文本描述出发，我们首先构建捕捉环境几何结构（墙壁、地板等）的网格，随后通过图像合成、分割与对象重建技术，在网格结构中填充布局真实的对象。该网格支架随后被渲染以条件化图像合成，为一致的外观生成提供结构骨架。这使得我们能够生成可扩展、任意尺寸且对象丰富多样的3D场景，在保持稳健3D一致性的同时实现照片级细节。我们相信这标志着向生成真正环境规模、沉浸式3D世界迈出了重要一步。
URL
https://arxiv.org/abs/2603.22972
