Abstract
Learning manipulation skills from human demonstration videos presents a promising yet challenging problem, primarily due to the significant embodiment gap between human body and robot manipulators. Existing methods rely on paired datasets or hand-crafted rewards, which limit scalability and generalization. We propose TrajSkill, a framework for Trajectory Conditioned Cross-embodiment Skill Transfer, enabling robots to acquire manipulation skills directly from human demonstration videos. Our key insight is to represent human motions as sparse optical flow trajectories, which serve as embodiment-agnostic motion cues by removing morphological variations while preserving essential dynamics. Conditioned on these trajectories together with visual and textual inputs, TrajSkill jointly synthesizes temporally consistent robot manipulation videos and translates them into executable actions, thereby achieving cross-embodiment skill transfer. Extensive experiments are conducted, and the results on simulation data (MetaWorld) show that TrajSkill reduces FVD by 39.6\% and KVD by 36.6\% compared with the state-of-the-art, and improves cross-embodiment success rate by up to 16.7\%. Real-robot experiments in kitchen manipulation tasks further validate the effectiveness of our approach, demonstrating practical human-to-robot skill transfer across embodiments.
Abstract (translated)
从人类演示视频中学习操作技能是一个有前景但又极具挑战性的问题，主要原因是人手和机器人执行器之间存在显著的身体差异。现有的方法依赖于成对的数据集或手工制作的奖励机制，这限制了可扩展性和泛化能力。我们提出了一个名为TrajSkill的框架，这是一种基于轨迹条件跨体形技能转移的方法，能够使机器人直接从人类演示视频中学习操作技能。 我们的核心观点是将人类的动作表示为稀疏光流轨迹，这些轨迹通过消除形态变化而保留了关键的动力学特征，并作为与具体身体形式无关的动作线索。在这些轨迹、视觉和文本输入的条件下，TrajSkill可以同时生成时间上一致的机器人操作视频，并将其转化为可执行的操作动作，从而实现跨体形技能转移。 我们进行了广泛的实验，在模拟数据（MetaWorld）上的结果显示，TrajSkill相比现有最佳方法减少了39.6%的FVD分数和36.6%的KVD分数，并将跨身体形式的成功率提高了高达16.7%。在厨房操作任务中的真实机器人实验进一步验证了我们方法的有效性，展示了从人类到机器人的实际技能转移能力，能够跨越不同的身体形式。
URL
https://arxiv.org/abs/2510.07773
