Abstract
For robots to move in the real world, they must first correctly understand the state of its own body and the tools that it holds. In this research, we propose DIJE, an algorithm to estimate the image Jacobian for every pixel. It is based on an optical flow calculation and a simplified Kalman Filter that can be efficiently run on the whole image in real time. It does not rely on markers nor knowledge of the robotic structure. We use the DIJE in a self-recognition process which can robustly distinguish between movement by the robot and by external entities, even when the motion overlaps. We also propose a visual servoing controller based on DIJE, which can learn to control the robot's body to conduct reaching movements or bimanual tool-tip control. The proposed algorithms were implemented on a physical musculoskeletal robot and its performance was verified. We believe that such global estimation of the visuomotor policy has the potential to be extended into a more general framework for manipulation.
Abstract (translated)
为了使机器人能够在真实世界中移动，它们必须首先正确理解自身状态及其持有的工具的状态。在这项研究中，我们提出了一种名为DIJE的算法，该算法用于估计每个像素的图像雅可比矩阵。DIJE基于光学流计算和一个简化的卡尔曼滤波器，后者可以实时高效地在整个图像上运行。它不需要标记物也不依赖于机器人的结构知识。我们在自我识别过程中使用了DIJE，这使得机器人能够稳健地区分自身运动与其他外部实体的运动，即使这些运动重叠时也能区分清楚。我们还提出了一种基于DIJE的视觉伺服控制器，该控制器可以学习控制机器人的身体以执行抓取动作或双臂工具尖端的操作控制。所提出的算法在物理肌骨机器人上进行了实现，并对其性能进行了验证。我们认为这种全局化的视动策略估计方法具有潜在可能被扩展为更通用的手部操作框架的基础。
URL
https://arxiv.org/abs/2507.00446
