Abstract
Creating high-fidelity 3D models of indoor environments is essential for applications in design, virtual reality, and robotics. However, manual 3D modeling remains time-consuming and labor-intensive. While recent advances in generative AI have enabled automated scene synthesis, existing methods often face challenges in balancing visual quality, diversity, semantic consistency, and user control. A major bottleneck is the lack of a large-scale, high-quality dataset tailored to this task. To address this gap, we introduce a comprehensive synthetic dataset, featuring 12,328 structured annotated scenes with 57,440 rooms, and 4.7M photorealistic 2D renderings. Leveraging this dataset, we present SpatialGen, a novel multi-view multi-modal diffusion model that generates realistic and semantically consistent 3D indoor scenes. Given a 3D layout and a reference image (derived from a text prompt), our model synthesizes appearance (color image), geometry (scene coordinate map), and semantic (semantic segmentation map) from arbitrary viewpoints, while preserving spatial consistency across modalities. SpatialGen consistently generates superior results to previous methods in our experiments. We are open-sourcing our data and models to empower the community and advance the field of indoor scene understanding and generation.
Abstract (translated)
创建高保真度的室内环境3D模型对于设计、虚拟现实和机器人技术的应用至关重要。然而，手动进行3D建模仍然耗时且劳动密集型。虽然最近在生成式AI方面的进展已经使得场景自动化合成成为可能，但现有的方法往往面临着平衡视觉质量、多样性、语义一致性以及用户控制的挑战。一个重要瓶颈在于缺少一个大规模高质量的数据集来专门应对这一任务。为解决这一缺口，我们介绍了一个全面的人工数据集，该数据集中包含了12,328个结构化标注场景和57,440个房间，并有4.7M张逼真的2D渲染图像。利用这一数据集，我们提出了一种新颖的多视角多模态扩散模型——SpatialGen，能够生成视觉上真实且语义一致的室内三维场景。给定一个3D布局和一张参考图片（由文本提示派生），我们的模型可以从任意视角合成立体感外观（颜色图像）、几何结构（场景坐标图）以及语义信息（语义分割图），同时在不同模态间保持空间一致性。在实验中，SpatialGen始终比先前的方法产生更好的结果。我们计划开放源代码并公开数据和模型以赋能社区，并推动室内场景理解和生成领域的发展。
URL
https://arxiv.org/abs/2509.14981
