Abstract
Natural language interfaces to embodied AI are becoming more ubiquitous in our daily lives. This opens further opportunities for language-based interaction with embodied agents, such as a user instructing an agent to execute some task in a specific location. For example, "put the bowls back in the cupboard next to the fridge" or "meet me at the intersection under the red sign." As such, we need methods that interface between natural language and map representations of the environment. To this end, we explore the question of whether we can use an open-set natural language query to identify a scene represented by a 3D scene graph. We define this task as "language-based scene-retrieval" and it is closely related to "coarse-localization," but we are instead searching for a match from a collection of disjoint scenes and not necessarily a large-scale continuous map. Therefore, we present Text2SceneGraphMatcher, a "scene-retrieval" pipeline that learns joint embeddings between text descriptions and scene graphs to determine if they are matched. The code, trained models, and datasets will be made public.
Abstract (translated)
自然语言界面与 embodied 人工智能越来越普及地出现在我们日常生活中。这为基于语言与嵌入式代理的交互提供了更多的机会，例如用户指示代理在特定位置执行一些任务。例如，"把碗放进冰箱里的餐柜里" 或 "在红标志下路口见面"。因此，我们需要方法在自然语言和环境地图表示之间进行交互。为此，我们探讨了是否可以使用 open-set 自然语言查询来确定由 3D 场景图表示的场景。我们将这个任务定义为 "基于语言的场景检索"，它与 "粗定位" 密切相关，但我们实际上在寻找一个匹配从一系列不相交的场景集合中，而不是一个大规模连续地图。因此，我们提出了 Text2SceneGraphMatcher，一个 "场景检索" 管道，它学会了文本描述和场景图之间的联合嵌入，以确定它们是否匹配。代码，训练的模型和数据集将公开发布。
URL
https://arxiv.org/abs/2404.14565
