Abstract
Language-instructed robot manipulation has garnered significant interest due to the potential of learning from collected data. While the challenges in high-level perception and planning are continually addressed along the progress of general large pre-trained models, the low precision of low-level action estimation has emerged as the key limiting factor in manipulation performance. To this end, this paper introduces a novel robot manipulation framework, i.e., ActionSink, to pave the way toward precise action estimations in the field of learning-based robot manipulation. As the name suggests, ActionSink reformulates the actions of robots as action-caused optical flows from videos, called "action flow", in a self-supervised manner, which are then used to be retrieved and integrated to enhance the action estimation. Specifically, ActionSink incorporates two primary modules. The first module is a coarse-to-fine action flow matcher, which continuously refines the accuracy of action flow via iterative retrieval and denoising process. The second module is a dynamic action flow integrator, which employs a working memory pool that dynamically and efficiently manages the historical action flows that should be used to integrate to enhance the current action estimation. In this module, a multi-layer fusion module is proposed to integrate direct estimation and action flows from both the current and the working memory, achieving highly accurate action estimation through a series of estimation-integration processes. Our ActionSink framework outperformed prior SOTA on the LIBERO benchmark by a 7.9\% success rate, and obtained nearly an 8\% accuracy gain on the challenging long-horizon visual task LIBERO-Long.
Abstract (translated)
基于指令的语言引导机器人操作由于可以从收集的数据中学习而引起了广泛的关注。尽管随着通用大型预训练模型的进步，高级感知和规划方面的挑战正在不断得到解决，但低级动作估计的精度不高已成为操控性能的主要限制因素。为此，本文提出了一种新的机器人操作框架——ActionSink，旨在推动基于学习的机器人操纵领域精确动作估算的发展。 正如其名称所暗示的那样，ActionSink 通过自监督的方式将机器人的动作重新表述为视频中由动作引起的光流（称为“动作流”），然后将其检索并整合以增强动作估计。具体来说，ActionSink 包含两个主要模块：一个是从粗到细的动作流匹配器和一个动态的动作流集成器。 第一个模块是一个从粗到细的动作流匹配器，通过迭代检索和去噪过程不断细化动作流的精度。第二个模块是动态动作流集成器，它使用一个工作内存池来动态且高效地管理需要用于当前动作估算的历史动作流。在这个模块中，提出了一个多层融合模块以整合直接估计和来自当前以及工作内存的动作流，通过一系列估计-集成过程实现了高度准确的动作估计。 我们的ActionSink框架在LIBERO基准测试中比之前的最先进方法（SOTA）高出7.9%的成功率，并且在具有挑战性的长时间视觉任务LIBERO-Long上获得了近8%的准确性提升。
URL
https://arxiv.org/abs/2508.03218
