Abstract
We show that off-the-shelf text-based Transformers, with no additional training, can perform few-shot in-context visual imitation learning, mapping visual observations to action sequences that emulate the demonstrator's behaviour. We achieve this by transforming visual observations (inputs) and trajectories of actions (outputs) into sequences of tokens that a text-pretrained Transformer (GPT-4 Turbo) can ingest and generate, via a framework we call Keypoint Action Tokens (KAT). Despite being trained only on language, we show that these Transformers excel at translating tokenised visual keypoint observations into action trajectories, performing on par or better than state-of-the-art imitation learning (diffusion policies) in the low-data regime on a suite of real-world, everyday tasks. Rather than operating in the language domain as is typical, KAT leverages text-based Transformers to operate in the vision and action domains to learn general patterns in demonstration data for highly efficient imitation learning, indicating promising new avenues for repurposing natural language models for embodied tasks. Videos are available at this https URL.
Abstract (translated)
我们证明了，无需额外训练，基于文本的Transformer模型可以实现少样本的上下文视觉模仿学习，将视觉观察映射到模仿者行为的动作序列。我们通过一个我们称之为键点动作词（KAT）的框架实现了这一点。尽管这些模型仅在语言领域训练，但我们证明了这些Transformer在将标记的视觉关键点观察映射为动作序列方面表现出色，在低数据情况下与状态最先进的模仿学习（扩散策略）相当或者更好。与典型的在语言域操作不同，KAT利用基于文本的Transformer在视觉和动作域操作，以学习演示数据中高度有效的模仿学习，表明了将自然语言模型用于 embodied 任务的新的途径。视频可在此处访问：https://www.youtube.com/watch?v=uRstRQZ0Q7g。
URL
https://arxiv.org/abs/2403.19578
