Abstract
3D scene generation plays a crucial role in gaming, artistic creation, virtual reality and many other domains. However, current 3D scene design still relies heavily on extensive manual effort from creators, and existing automated methods struggle to generate open-domain scenes or support flexible editing. As a result, generating 3D worlds directly from text has garnered increasing attention. In this paper, we introduce HOLODECK 2.0, an advanced vision-language-guided framework for 3D world generation with support for interactive scene editing based on human feedback. HOLODECK 2.0 can generate diverse and stylistically rich 3D scenes (e.g., realistic, cartoon, anime, and cyberpunk styles) that exhibit high semantic fidelity to fine-grained input descriptions, suitable for both indoor and open-domain environments. HOLODECK 2.0 leverages vision-language models (VLMs) to identify and parse the objects required in a scene and generates corresponding high-quality assets via state-of-the-art 3D generative models. It then iteratively applies spatial constraints derived from the VLMs to achieve semantically coherent and physically plausible layouts. Human evaluations and CLIP-based assessments demonstrate that HOLODECK 2.0 effectively generates high-quality scenes closely aligned with detailed textual descriptions, consistently outperforming baselines across indoor and open-domain scenarios. Additionally, we provide editing capabilities that flexibly adapt to human feedback, supporting layout refinement and style-consistent object edits. Finally, we present a practical application of HOLODECK 2.0 in procedural game modeling, generating visually rich and immersive environments, potentially boosting efficiency.
Abstract (translated)
3D场景生成在游戏、艺术创作、虚拟现实等多个领域中扮演着至关重要的角色。然而，当前的3D场景设计仍然高度依赖于创作者大量的手工劳动，并且现有的自动化方法难以生成开放领域的场景或支持灵活编辑。因此，从文本直接生成3D世界引起了越来越多的关注。在这篇论文中，我们介绍了HOLODECK 2.0，这是一个先进的基于视觉-语言指导的框架，用于生成具有互动场景编辑功能（基于人类反馈）的3D世界。HOLODECK 2.0能够生成多样化且风格丰富的3D场景（例如：写实、卡通、动漫和赛博朋克风格），这些场景与细粒度输入描述的高度语义一致，适用于室内及开放领域环境。通过利用视觉-语言模型(VLMs)，HOLODECK 2.0可以识别并解析场景所需的对象，并借助最先进的3D生成模型来生成相应的高质量资产。然后，它会迭代应用由VLMs推导的空间约束，以实现语义连贯且物理上合理的布局。人类评估和CLIP（对比语言-图像预训练）评估表明，HOLODECK 2.0能够有效生成与详细文本描述高度一致的高质量场景，在室内及开放领域场景中持续优于基准方法。此外，我们提供了灵活适应人类反馈的编辑功能，支持布局调整和风格一致性对象编辑。最后，我们将展示HOLODECK 2.0在程序化游戏建模中的实际应用，生成视觉丰富且沉浸式的环境，有望提高效率。
URL
https://arxiv.org/abs/2508.05899
