Abstract
Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: this https URL
Abstract (translated)
从日常人类视频中学习操作技能的机器人可以无需繁琐的数据收集过程就能获得广泛的能力。我们提出了一种视频到视频转换框架，该框架能够将普通的“人-物”互动视频转化为带有真实且物理上合理的机器人操作动作的视频。我们的方法在训练时不需要配对的人类和机器人的视频数据，仅需要一组未配对的机器人视频，使得系统易于扩展。 我们引入了一种可转移表示形式，以弥合实体差距：通过在训练视频中绘制机械臂来获得干净的背景，并叠加一个简单的视觉线索（标记物和指示夹持器位置和方向的箭头），我们可以使生成模型有条件地将机械臂重新插入场景。在测试阶段，我们对人类视频执行同样的过程（擦除人物并叠加人体姿态提示），从而能够产生高质量的机器人视频，这些视频模仿了人类的动作。 我们将最先进的视频扩散模型（Wan 2.2）以上下文学习的方式进行微调，确保其时间一致性，并充分利用其中丰富的先验知识。实证结果表明，与基线方法相比，我们的方法实现了更真实和基于物理的操作动作，预示着从未标记的人类视频中大规模训练机器人具有广阔前景。 项目主页：[此链接](https://this-url.com)
URL
https://arxiv.org/abs/2512.09406
