Abstract
Creating large-scale interactive 3D environments is essential for the development of Robotics and Embodied AI research. Current methods, including manual design, procedural generation, diffusion-based scene generation, and large language model (LLM) guided scene design, are hindered by limitations such as excessive human effort, reliance on predefined rules or training datasets, and limited 3D spatial reasoning ability. Since pre-trained 2D image generative models better capture scene and object configuration than LLMs, we address these challenges by introducing Architect, a generative framework that creates complex and realistic 3D embodied environments leveraging diffusion-based 2D image inpainting. In detail, we utilize foundation visual perception models to obtain each generated object from the image and leverage pre-trained depth estimation models to lift the generated 2D image to 3D space. Our pipeline is further extended to a hierarchical and iterative inpainting process to continuously generate placement of large furniture and small objects to enrich the scene. This iterative structure brings the flexibility for our method to generate or refine scenes from various starting points, such as text, floor plans, or pre-arranged environments.
Abstract (translated)
创建大规模交互式3D环境对于机器人技术和具身人工智能研究的发展至关重要。当前的方法，包括手动设计、程序生成、基于扩散的场景生成以及大型语言模型（LLM）引导的场景设计，都受到过度依赖人力投入、对预定义规则或训练数据集的依赖性以及有限的3D空间推理能力等限制的影响。鉴于预先训练的2D图像生成模型比LLM更能捕捉到场景和物体配置的特点，我们通过引入Architect解决了这些挑战——这是一种利用基于扩散的2D图像修复来创建复杂且逼真的3D具身环境的生成框架。具体来说，我们使用基础视觉感知模型从图像中获取每个生成的对象，并借助预先训练好的深度估计模型将生成的2D图像提升到3D空间。我们的流程进一步扩展为一个分层和迭代的修复过程，以持续生成大型家具和小型物体的位置来丰富场景。这种迭代结构带来了灵活性，使我们的方法能够从各种起点（如文本、平面图或预先布置的环境）开始生成或细化场景。
URL
https://arxiv.org/abs/2411.09823
