Abstract
Learning a generalizable bimanual manipulation policy is extremely challenging for embodied agents due to the large action space and the need for coordinated arm movements. Existing approaches rely on Vision-Language-Action (VLA) models to acquire bimanual policies. However, transferring knowledge from single-arm datasets or pre-trained VLA models often fails to generalize effectively, primarily due to the scarcity of bimanual data and the fundamental differences between single-arm and bimanual manipulation. In this paper, we propose a novel bimanual foundation policy by fine-tuning the leading text-to-video models to predict robot trajectories and training a lightweight diffusion policy for action generation. Given the lack of embodied knowledge in text-to-video models, we introduce a two-stage paradigm that fine-tunes independent text-to-flow and flow-to-video models derived from a pre-trained text-to-video model. Specifically, optical flow serves as an intermediate variable, providing a concise representation of subtle movements between images. The text-to-flow model predicts optical flow to concretize the intent of language instructions, and the flow-to-video model leverages this flow for fine-grained video prediction. Our method mitigates the ambiguity of language in single-stage text-to-video prediction and significantly reduces the robot-data requirement by avoiding direct use of low-level actions. In experiments, we collect high-quality manipulation data for real dual-arm robot, and the results of simulation and real-world experiments demonstrate the effectiveness of our method.
Abstract (translated)
学习一种通用的双臂操作策略对于具身代理来说极为具有挑战性，原因在于庞大的动作空间以及需要协调的手臂运动。现有的方法依赖于视觉-语言-行动（VLA）模型来获取双臂策略。然而，从单臂数据集或预训练的VLA模型中转移知识往往难以有效泛化，主要是由于双臂数据稀缺和单臂与双臂操作之间存在本质差异所致。 在本文中，我们提出了一种新颖的双臂基础策略，通过微调领先的文本到视频模型来预测机器人轨迹，并训练一个轻量级扩散策略用于动作生成。鉴于文本到视频模型缺乏具身知识，我们引入了一个两阶段范式，以预训练的文本到视频模型为基础，对独立的文本到流（flow）和流到视频模型进行微调。具体来说，光学流动充当中间变量，提供图像之间细微运动的简洁表示。文本到流模型预测光学流动，将语言指令的具体意图形式化；而流到视频模型则利用此流动实现精细的视频预测。我们的方法减轻了一步式文本到视频预测中语言模糊性的问题，并通过避免直接使用低级动作显著减少了对机器人数据的需求。 在实验中，我们为实际的双臂机器人收集了高质量的操作数据，模拟和真实世界实验的结果展示了该方法的有效性。
URL
https://arxiv.org/abs/2505.24156
