Abstract
Robots often struggle to follow free-form human instructions in real-world settings due to computational and sensing limitations. We address this gap with a lightweight, fully on-device pipeline that converts natural-language commands into reliable manipulation. Our approach has two stages: (i) the instruction to actions module (Instruct2Act), a compact BiLSTM with a multi-head-attention autoencoder that parses an instruction into an ordered sequence of atomic actions (e.g., reach, grasp, move, place); and (ii) the robot action network (RAN), which uses the dynamic adaptive trajectory radial network (DATRN) together with a vision-based environment analyzer (YOLOv8) to generate precise control trajectories for each sub-action. The entire system runs on a modest system with no cloud services. On our custom proprietary dataset, Instruct2Act attains 91.5% sub-actions prediction accuracy while retaining a small footprint. Real-robot evaluations across four tasks (pick-place, pick-pour, wipe, and pick-give) yield an overall 90% success; sub-action inference completes in < 3.8s, with end-to-end executions in 30-60s depending on task complexity. These results demonstrate that fine-grained instruction-to-action parsing, coupled with DATRN-based trajectory generation and vision-guided grounding, provides a practical path to deterministic, real-time manipulation in resource-constrained, single-camera settings.
Abstract (translated)
机器人在现实环境中往往难以根据自由形式的人类指令执行任务，这主要是由于计算能力和感知能力的限制。我们提出了一种轻量级、完全在设备上运行的管道系统，可以将自然语言命令转换为可靠的操纵动作。我们的方法包括两个阶段： (i) 指令到动作模块（Instruct2Act）：这是一个紧凑型BiLSTM模型结合多头注意力自动编码器，能够解析指令，并将其转化为一系列原子动作（例如，到达、抓取、移动和放置）的有序序列。 (ii) 机器人行动网络（RAN）：该阶段使用动态自适应轨迹径向网络（DATRN）与基于视觉环境分析器（YOLOv8）共同工作，为每个子动作生成精确的控制轨迹。 整个系统在没有任何云端服务支持的情况下就可以在一个相对简单的小型设备上运行。在我们定制的专有数据集上，Instruct2Act在保持较小存储需求的同时，实现了91.5%的子动作预测准确率。实机评估显示，在四个任务（取放、倒水、擦拭和递送）中，成功率总体达到了90%，每个子行动推断时间少于3.8秒，而整个操作执行的时间则根据任务复杂度在30到60秒之间。 这些结果表明，细粒度的指令解析与DATRN基础轨迹生成以及视觉引导定位相结合的方法，在资源受限、单摄像头设置中提供了确定性实时操控实现的实际路径。
URL
https://arxiv.org/abs/2602.09940
