Abstract
We address language-conditioned robotic manipulation using flow-based trajectory generation, which enables training on human and web videos of object manipulation and requires only minimal embodiment-specific data. This task is challenging, as object trajectory generation from pre-manipulation images and natural language instructions requires appropriate instruction-flow alignment. To tackle this challenge, we propose the flow-based Language Instruction-guided open-Loop ACtion generator (LILAC). This flow-based Vision-Language-Action model (VLA) generates object-centric 2D optical flow from an RGB image and a natural language instruction, and converts the flow into a 6-DoF manipulator trajectory. LILAC incorporates two key components: Semantic Alignment Loss, which strengthens language conditioning to generate instruction-aligned optical flow, and Prompt-Conditioned Cross-Modal Adapter, which aligns learned visual prompts with image and text features to provide rich cues for flow generation. Experimentally, our method outperformed existing approaches in generated flow quality across multiple benchmarks. Furthermore, in physical object manipulation experiments using free-form instructions, LILAC demonstrated a superior task success rate compared to existing methods. The project page is available at this https URL.
Abstract (translated)
我们采用基于流式的轨迹生成方法来解决语言条件机器人操作问题，该方法支持利用人类操作和网络视频进行训练，且仅需极少量的具身数据。由于需要根据操作前图像和自然语言指令生成物体轨迹，并确保指令与流场的对齐，该任务具有挑战性。为应对这一挑战，我们提出了基于流式的语言指令引导开环动作生成器（LILAC）。这一基于流式的视觉-语言-动作模型（VLA）可从RGB图像和自然语言指令中生成以物体为中心的二维光流，并将光流转换为6自由度机械臂轨迹。LILAC包含两个关键组件：语义对齐损失（用于强化语言条件以生成与指令对齐的光流）和提示条件跨模态适配器（用于将学习到的视觉提示与图像及文本特征对齐，为流场生成提供丰富线索）。实验表明，我们的方法在多个基准测试中生成的流场质量优于现有方法。此外，在使用自由形式指令的物理物体操作实验中，LILAC的任务成功率也显著高于现有方法。项目页面可通过此链接访问。
URL
https://arxiv.org/abs/2603.25481
