Abstract
Text-to-3D scene generation from natural language is highly desirable for digital content creation. However, existing methods are largely domain-restricted or reliant on predefined spatial relationships, limiting their capacity for unconstrained, open-vocabulary 3D scene synthesis. In this paper, we introduce SceneAssistant, a visual-feedback-driven agent designed for open-vocabulary 3D scene generation. Our framework leverages modern 3D object generation model along with the spatial reasoning and planning capabilities of Vision-Language Models (VLMs). To enable open-vocabulary scene composition, we provide the VLMs with a comprehensive set of atomic operations (e.g., Scale, Rotate, FocusOn). At each interaction step, the VLM receives rendered visual feedback and takes actions accordingly, iteratively refining the scene to achieve more coherent spatial arrangements and better alignment with the input text. Experimental results demonstrate that our method can generate diverse, open-vocabulary, and high-quality 3D scenes. Both qualitative analysis and quantitative human evaluations demonstrate the superiority of our approach over existing methods. Furthermore, our method allows users to instruct the agent to edit existing scenes based on natural language commands. Our code is available at this https URL
Abstract (translated)
从文本生成三维场景是数字内容创作中非常有价值的需求。然而，现有的方法大多局限于特定领域或者依赖于预定义的空间关系，这限制了它们在不受限、开放式词汇表3D场景合成中的应用能力。在这篇论文中，我们介绍了SceneAssistant，这是一个专为开放词汇表的3D场景生成设计的视觉反馈驱动代理。我们的框架结合了现代三维对象生成模型以及Vision-Language Models（VLMs）的空间推理和规划功能。为了支持开放式词汇表的场景组合，我们为VLM提供了全面的基础操作集（例如：缩放、旋转、聚焦等）。在每个交互步骤中，当接收到渲染后的视觉反馈时，VLM会相应地采取行动，并迭代优化场景以实现更连贯的空间排列和更好地与输入文本对齐。实验结果表明，我们的方法可以生成多样化的、开放式词汇表的以及高质量的3D场景。无论是定性分析还是定量的人类评估都证明了我们这种方法优于现有技术。此外，我们的方法允许用户根据自然语言命令指示代理编辑现有的场景。我们的代码可以在提供的网址上获取。 这种翻译旨在概述该论文的主要贡献和技术细节，并且提供了一个对非专业读者友好的版本。
URL
https://arxiv.org/abs/2603.12238
