Abstract
Large Vision-Language Models (VLMs), such as GPT-4, have achieved remarkable success across various fields. However, there are few studies on 3D indoor scene generation with VLMs. This paper considers this task as a planning problem subject to spatial and layout common sense constraints. To solve the problem with a VLM, we propose a new global-local tree search algorithm. Globally, the method places each object sequentially and explores multiple placements during each placement process, where the problem space is represented as a tree. To reduce the depth of the tree, we decompose the scene structure hierarchically, i.e. room level, region level, floor object level, and supported object level. The algorithm independently generates the floor objects in different regions and supported objects placed on different floor objects. Locally, we also decompose the sub-task, the placement of each object, into multiple steps. The algorithm searches the tree of problem space. To leverage the VLM model to produce positions of objects, we discretize the top-down view space as a dense grid and fill each cell with diverse emojis to make to cells distinct. We prompt the VLM with the emoji grid and the VLM produces a reasonable location for the object by describing the position with the name of emojis. The quantitative and qualitative experimental results illustrate our approach generates more plausible 3D scenes than state-of-the-art approaches. Our source code is available at this https URL .
Abstract (translated)
大型视觉-语言模型（VLMs），如GPT-4，在多个领域取得了显著的成功。然而，关于使用VLM生成三维室内场景的研究相对较少。本文将这一任务视为一个受空间和布局常识约束的规划问题，并提出了一种新的全局局部树搜索算法来解决该问题。 在全局层面上，本方法依次放置每个对象并探索每次放置过程中的多种放置方式。为了减少问题空间表示为树结构的深度，我们将场景结构分层分解，即按房间层级、区域层级、楼层物体层级和支持物体层级进行拆解。算法独立生成不同区域中的地面物体和不同地面物体上的支撑物。 在局部层面，我们将每个对象的放置任务进一步细分为多个步骤，并通过探索问题空间的树形图来搜索最优解。为了利用VLM模型产生物体的位置信息，我们把俯视视角的空间离散化为密集网格，并向每个格子填充多样化的表情符号以使其独特可辨认。然后用带有表情符号网格的提示询问VLM，从而得到由表情名称描述的一个合理位置。 通过量化和定性的实验结果表明，我们的方法能够生成比现有最新技术更合理的三维场景。我们的源代码可在提供的链接处获取。
URL
https://arxiv.org/abs/2503.18476
