Abstract
Learning from videos offers a promising path toward generalist robots by providing rich visual and temporal priors beyond what real robot datasets contain. While existing video generative models produce impressive visual predictions, they are difficult to translate into low-level actions. Conversely, latent-action models better align videos with actions, but they typically operate at the single-step level and lack high-level planning capabilities. We bridge this gap by introducing Skill Abstraction from Optical Flow (SOF), a framework that learns latent skills from large collections of action-free videos. Our key idea is to learn a latent skill space through an intermediate representation based on optical flow that captures motion information aligned with both video dynamics and robot actions. By learning skills in this flow-based latent space, SOF enables high-level planning over video-derived skills and allows for easier translation of these skills into actions. Experiments show that our approach consistently improves performance in both multitask and long-horizon settings, demonstrating the ability to acquire and compose skills directly from raw visual data.
Abstract (translated)
从视频中学习为通用机器人的开发提供了一条充满希望的道路，因为它提供了超出真实机器人数据集的丰富视觉和时间先验。尽管现有的视频生成模型能够产生令人印象深刻的视觉预测，但它们很难转化为低级别的行动指令。相反，基于潜在动作的模型更好地将视频与具体操作对齐，不过通常这些模型只在单步水平上运行，并且缺乏高级规划能力。为了弥合这一差距，我们引入了Skill Abstraction from Optical Flow (SOF)框架，该框架能够从大量的无动作视频集合中学习潜在技能。我们的核心理念是通过基于光学流的中间表示来学习潜在的技能空间，这种表示方法能捕捉与视频动态和机器人操作都对齐的运动信息。通过对这种基于流动的潜在空间中的技能进行学习，SOF能够在基于视频获取的技能上实现高级规划，并使这些技能更容易转化为具体行动。实验表明，我们的方法在多任务和长时展望设置中均表现出持续改进性能的能力，展示了直接从原始视觉数据中获得和组合技能的能力。
URL
https://arxiv.org/abs/2512.20052
