Abstract
Robot manipulation research still suffers from significant data scarcity: even the largest robot datasets are orders of magnitude smaller and less diverse than those that fueled recent breakthroughs in language and vision. We introduce Masquerade, a method that edits in-the-wild egocentric human videos to bridge the visual embodiment gap between humans and robots and then learns a robot policy with these edited videos. Our pipeline turns each human video into robotized demonstrations by (i) estimating 3-D hand poses, (ii) inpainting the human arms, and (iii) overlaying a rendered bimanual robot that tracks the recovered end-effector trajectories. Pre-training a visual encoder to predict future 2-D robot keypoints on 675K frames of these edited clips, and continuing that auxiliary loss while fine-tuning a diffusion policy head on only 50 robot demonstrations per task, yields policies that generalize significantly better than prior work. On three long-horizon, bimanual kitchen tasks evaluated in three unseen scenes each, Masquerade outperforms baselines by 5-6x. Ablations show that both the robot overlay and co-training are indispensable, and performance scales logarithmically with the amount of edited human video. These results demonstrate that explicitly closing the visual embodiment gap unlocks a vast, readily available source of data from human videos that can be used to improve robot policies.
Abstract (translated)
机器人操纵研究仍然受到数据稀缺的严重影响：即使是最大的机器人数据集，也比最近语言和视觉领域突破所使用的数据小得多且多样性不足。我们引入了一种名为Masquerade的方法，该方法通过编辑野外的第一人称人类视频来弥合人类与机器人的视觉表现差距，并利用这些编辑后的视频学习机器人策略。我们的流水线将每个第一人称人类视频转换为机器人演示，具体步骤如下：(i) 估计3D手部姿态；(ii) 给人类手臂补帧；以及(iii) 在视频中叠加一个渲染的双臂机器人以跟踪恢复出来的末端执行器轨迹。通过对675,000帧编辑后的片段进行预训练，使视觉编码器预测未来2D机器人的关键点，并在仅使用每项任务50个机器人演示的情况下继续微调扩散策略头，在辅助损失下进行训练，可以得到比以往研究显著更好的泛化策略。在三个具有长时域和双臂厨房任务的三个未见过场景中，Masquerade的表现优于基线方法多达5-6倍。消融研究表明，机器人的叠加以及协同训练都是必不可少的，并且性能随着编辑后的人类视频数量以对数方式增加。这些结果表明，明确地弥合视觉表现差距可以解锁大量现成可用的数据源，即人类视频中的数据，这可用于改善机器人策略。
URL
https://arxiv.org/abs/2508.09976
