Abstract
Object navigation in unknown environments is crucial for deploying embodied agents in real-world applications. While we have witnessed huge progress due to large-scale scene datasets, faster simulators, and stronger models, previous studies mainly focus on limited scene types and target objects. In this paper, we study a new task of navigating to diverse target objects in a large number of scene types. To benchmark the problem, we present a large-scale scene dataset, DivScene, which contains 4,614 scenes across 81 different types. With the dataset, we build an end-to-end embodied agent, NatVLM, by fine-tuning a Large Vision Language Model (LVLM) through imitation learning. The LVLM is trained to take previous observations from the environment and generate the next actions. We also introduce CoT explanation traces of the action prediction for better performance when tuning LVLMs. Our extensive experiments find that we can build a performant LVLM-based agent through imitation learning on the shortest paths constructed by a BFS planner without any human supervision. Our agent achieves a success rate that surpasses GPT-4o by over 20%. Meanwhile, we carry out various analyses showing the generalization ability of our agent.
Abstract (translated)
在未知环境中进行物体导航对于在现实应用中部署感知代理至关重要。虽然我们通过大规模场景数据、高速模拟器和更强的模型见证了巨大的进步，但之前的研究主要集中在有限的场景类型和目标对象上。在本文中，我们研究了在多种场景类型中导航到多个目标对象的新任务。为了进行基准测试，我们提出了一个大型场景数据集DivScene，包含81个不同类型的场景，总共有4,614个场景。通过这个数据集，我们在基于模仿学习对一个大视觉语言模型（LVLM）进行微调，以构建端到端的 embodied 代理，NatVLM。LVLM 通过从环境中获取先前的观察并生成下一个动作进行训练。我们还引入了动作预测的 CoT 轨迹，以便在调整 LVLMs 时获得更好的性能。通过广泛的实验，我们发现，在没有人类监督的情况下，通过模仿学习可以在最短路径上构建的 BFS 规划器上构建出高性能的 LVLM 代理。我们的代理在 GPT-4o 上的成功率超过了20%。同时，我们进行了各种分析，证明了我们的代理具有很强的泛化能力。
URL
https://arxiv.org/abs/2410.02730
