Abstract
Generating realistic 3D indoor scenes from user inputs remains a challenging problem in computer vision and graphics, requiring careful balance of geometric consistency, spatial relationships, and visual realism. While neural generation methods often produce repetitive elements due to limited global spatial reasoning, procedural approaches can leverage constraints for controllable generation but struggle with multi-constraint scenarios. When constraints become numerous, object collisions frequently occur, forcing the removal of furniture items and compromising layout completeness. To address these limitations, we propose RoomCraft, a multi-stage pipeline that converts real images, sketches, or text descriptions into coherent 3D indoor scenes. Our approach combines a scene generation pipeline with a constraint-driven optimization framework. The pipeline first extracts high-level scene information from user inputs and organizes it into a structured format containing room type, furniture items, and spatial relations. It then constructs a spatial relationship network to represent furniture arrangements and generates an optimized placement sequence using a heuristic-based depth-first search (HDFS) algorithm to ensure layout coherence. To handle complex multi-constraint scenarios, we introduce a unified constraint representation that processes both formal specifications and natural language inputs, enabling flexible constraint-oriented adjustments through a comprehensive action space design. Additionally, we propose a Conflict-Aware Positioning Strategy (CAPS) that dynamically adjusts placement weights to minimize furniture collisions and ensure layout completeness. Extensive experiments demonstrate that RoomCraft significantly outperforms existing methods in generating realistic, semantically coherent, and visually appealing room layouts across diverse input modalities.
Abstract (translated)
从用户输入生成逼真的3D室内场景在计算机视觉和图形学领域仍然是一个挑战性问题，需要精细地平衡几何一致性、空间关系以及视觉真实感。虽然神经网络生成方法常因全球空间推理能力有限而产生重复元素，程序化方法可以通过利用约束实现可控生成，但在面对多约束场景时却往往力不从心。当约束数量增多时，物体碰撞频繁发生，导致移除家具物品并损害布局完整性。为了克服这些限制，我们提出了RoomCraft，这是一种将真实图像、草图或文本描述转换为连贯3D室内场景的多阶段流水线方法。我们的方法结合了场景生成管道和基于约束驱动优化框架。 该管道首先从用户输入中提取高层次场景信息，并将其组织成包含房间类型、家具项目和空间关系的结构化格式。然后，它构建一个空间关系网络来表示家具布局安排，并使用启发式深度优先搜索（HDFS）算法生成最佳放置顺序以确保布局一致性。为了处理复杂的多约束场景，我们引入了一种统一的约束表示方法，可以同时处理正式规范和自然语言输入，并通过全面的动作设计实现灵活的基于约束调整。此外，我们还提出了一种冲突感知定位策略(CAPS)，该策略动态调整放置权重以尽量减少家具碰撞并确保布局完整性。 广泛的实验表明，RoomCraft在各种输入模式下显著优于现有方法，在生成逼真、语义连贯且视觉吸引人的房间布局方面表现出色。
URL
https://arxiv.org/abs/2506.22291
