Abstract
Recently, Vision Language Models (VLMs) have experienced significant advancements, yet these models still face challenges in spatial hierarchical reasoning within indoor scenes. In this study, we introduce ROOT, a VLM-based system designed to enhance the analysis of indoor scenes. Specifically, we first develop an iterative object perception algorithm using GPT-4V to detect object entities within indoor scenes. This is followed by employing vision foundation models to acquire additional meta-information about the scene, such as bounding boxes. Building on this foundational data, we propose a specialized VLM, SceneVLM, which is capable of generating spatial hierarchical scene graphs and providing distance information for objects within indoor environments. This information enhances our understanding of the spatial arrangement of indoor scenes. To train our SceneVLM, we collect over 610,000 images from various public indoor datasets and implement a scene data generation pipeline with a semi-automated technique to establish relationships and estimate distances among indoor objects. By utilizing this enriched data, we conduct various training recipes and finish SceneVLM. Our experiments demonstrate that \rootname facilitates indoor scene understanding and proves effective in diverse downstream applications, such as 3D scene generation and embodied AI. The code will be released at \url{this https URL}.
Abstract (translated)
最近，视觉语言模型（VLMs）取得了显著的进步，但这些模型在处理室内场景中的空间层次推理方面仍面临挑战。在这项研究中，我们介绍了ROOT——一个基于VLM的系统，旨在提高对室内场景的分析能力。具体来说，首先我们开发了一个迭代的对象感知算法，使用GPT-4V来检测室内场景中的对象实体。接着，利用视觉基础模型获取关于该场景的额外元信息，如边界框等。在此基础上，我们提出了一种专门的VLM——SceneVLM，它能够生成空间层次化的场景图，并提供室内环境中物体之间的距离信息。这些信息提升了我们对室内场景空间布局的理解。为了训练我们的SceneVLM，我们从各种公共的室内数据集中收集了超过610,000张图像，并采用半自动技术实现了一个场景数据生成管道来建立物体间的关系并估计它们的距离。通过利用丰富的数据，我们进行了多种训练方法并完成了SceneVLM的构建。实验表明，\rootname促进了对室内场景的理解，在各种下游应用中（如3D场景生成和具身AI）证明了其有效性。代码将在\url{这个 https URL}公开发布。
URL
https://arxiv.org/abs/2411.15714
