Abstract
Learning from Demonstration (LfD) offers a promising paradigm for robot skill acquisition. Recent approaches attempt to extract manipulation commands directly from video demonstrations, yet face two critical challenges: (1) general video captioning models prioritize global scene features over task-relevant objects, producing descriptions unsuitable for precise robotic execution, and (2) end-to-end architectures coupling visual understanding with policy learning require extensive paired datasets and struggle to generalize across objects and scenarios. To address these limitations, we propose a novel ``Human-to-Robot'' imitation learning pipeline that enables robots to acquire manipulation skills directly from unstructured video demonstrations, inspired by the human ability to learn by watching and imitating. Our key innovation is a modular framework that decouples the learning process into two distinct stages: (1) Video Understanding, which combines Temporal Shift Modules (TSM) with Vision-Language Models (VLMs) to extract actions and identify interacted objects, and (2) Robot Imitation, which employs TD3-based deep reinforcement learning to execute the demonstrated manipulations. We validated our approach in PyBullet simulation environments with a UR5e manipulator and in a real-world experiment with a UF850 manipulator across four fundamental actions: reach, pick, move, and put. For video understanding, our method achieves 89.97% action classification accuracy and BLEU-4 scores of 0.351 on standard objects and 0.265 on novel objects, representing improvements of 76.4% and 128.4% over the best baseline, respectively. For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations. The project website is available at this https URL.
Abstract (translated)
从演示学习（LfD）为机器人技能获取提供了一种有前景的范式。最近的方法试图直接从视频演示中提取操作命令，但面临着两个关键挑战：(1) 通用视频描述模型优先考虑全局场景特征而非与任务相关的对象，从而生成不适用于精确机器人执行的描述；(2) 将视觉理解与策略学习结合在一起的端到端架构需要大量的成对数据集，并且难以在不同物体和情景中泛化。为了解决这些局限性，我们提出了一种新型“人机”模仿学习流水线，使机器人能够直接从无结构视频演示中获取操作技能，灵感来自于人类通过观察和模仿来学习的能力。我们的关键创新是一个模块化的框架，将学习过程分解成两个独立的阶段：(1) 视频理解，结合时间偏移模块（TSM）与视觉-语言模型（VLM），以提取动作并识别交互对象；(2) 机器人模仿，使用基于TD3的深度强化学习来执行演示的操作。我们在PyBullet模拟环境中使用UR5e机械臂以及在现实世界实验中使用UF850机械臂进行了验证，涵盖了四个基本操作：到达、抓取、移动和放置。 对于视频理解，我们的方法达到了89.97%的动作分类准确率，并在标准对象上获得了BLEU-4得分为0.351，在新对象上的得分是0.265，分别比最佳基线提高了76.4% 和 128.4%。对于机器人操作，我们的框架在所有动作中达到了平均成功率87.5%，到达任务的成功率为100%，而复杂的抓放操作可达90%的最高成功率。 项目网站可在以下链接访问：[此处插入实际网址]（原文中的“this https URL”应替换为具体网址）。
URL
https://arxiv.org/abs/2602.19184
