Abstract
Generating controllable and interactive indoor scenes is fundamental to applications in game development, architectural visualization, and embodied AI training. Yet existing approaches either handle a narrow range of input modalities or rely on stochastic processes that hinder controllability. To overcome these limitations, we introduce RoomPilot, a unified framework that parses diverse multi-modal inputs--textual descriptions or CAD floor plans--into an Indoor Domain-Specific Language (IDSL) for indoor structured scene generation. The key insight is that a well-designed IDSL can act as a shared semantic representation, enabling coherent, high-quality scene synthesis from any single modality while maintaining interaction semantics. In contrast to conventional procedural methods that produce visually plausible but functionally inert layouts, RoomPilot leverages a curated dataset of interaction-annotated assets to synthesize environments exhibiting realistic object behaviors. Extensive experiments further validate its strong multi-modal understanding, fine-grained controllability in scene generation, and superior physical consistency and visual fidelity, marking a significant step toward general-purpose controllable 3D indoor scene generation.
Abstract (translated)
生成可控且互动的室内场景对于游戏开发、建筑可视化以及具身人工智能训练等应用来说至关重要。然而，现有的方法要么只能处理有限种类的输入模态（如文本描述或CAD平面图），要么依赖于随机过程来创建这些场景，这使得难以实现精确控制。为了克服这些问题，我们引入了RoomPilot——一个统一框架，它能够解析多种多样的多模态输入（包括文本描述和CAD楼层计划）并将其转换为室内专用领域的语言（IDSL），进而生成结构化的室内场景。关键的理念是：经过精心设计的IDSL可以作为共享语义表示，从单一模态中生成一致且高质量的场景，并保持交互性。 与传统的程序化方法不同，这些方法虽然能够产生视觉上可信但功能相对静止的布局，RoomPilot利用一个注有互动数据的数据集来合成表现出真实物体行为的环境。大量的实验进一步验证了它在多模态理解、场景生成中的精细控制以及物理一致性和视觉逼真度方面的卓越表现，标志着向着通用可控3D室内场景生成迈出了一大步。
URL
https://arxiv.org/abs/2512.11234
