Abstract
Programming robots by demonstration (PbD) is an intuitive concept, but scaling it to real-world variability remains a challenge for most current teaching frameworks. Conditional task graphs are very expressive and can be defined incrementally, which fits very well with the PbD idea. However, acting using conditional task graphs requires reliable perception-grounded online branch selection. In this paper, we present See & Switch, an interactive teaching-and-execution framework that represents tasks as user-extendable graphs of skill parts connected via decision states (DS), enabling conditional branching during replay. Unlike prior approaches that rely on manual branching or low-dimensional signals (e.g., proprioception), our vision-based Switcher uses eye-in-hand images (high-dimensional) to select among competing successor skill parts and to detect out-of-distribution contexts that require new demonstrations. We integrate kinesthetic teaching, joystick control, and hand gestures via an input-modality-abstraction layer and demonstrate that our proposed method is teaching modality-independent, enabling efficient in-situ recovery demonstrations. The system is validated in experiments on three challenging dexterous manipulation tasks. We evaluate our method under diverse conditions and furthermore conduct user studies with 8 participants. We show that the proposed method reliably performs branch selection and anomaly detection for novice users, achieving 90.7 % and 87.9 % accuracy, respectively, across 576 real-robot rollouts. We provide all code and data required to reproduce our experiments at this http URL.
Abstract (translated)
通过演示编程机器人（Programming by Demonstration，PbD）是一个直观的概念，但在现实中扩大其应用范围以适应各种变化仍然对大多数当前的教学框架构成了挑战。条件任务图非常具有表现力，并且可以逐步定义，这与PbD的理念十分契合。然而，在使用条件任务图时，需要可靠的基于感知的在线分支选择来执行相应的操作。 本文介绍了一种称为“See & Switch”的互动教学和执行框架，该框架将任务表示为通过决策状态（Decision States, DS）连接起来的技能部分图形，使用户能够在重播过程中实现条件分支。与以前依赖手动分支或低维信号（如本体感受）的方法不同，我们的基于视觉的“Switcher”使用手眼相机图像（高维度信息），来在竞争性的后续技能部分之间选择，并检测需要新演示的异常情况。 我们通过输入模式抽象层集成了助动式教学、操纵杆控制和手势操作。这种方法独立于教学模态，使用户能够有效地进行现场恢复性示范。该系统已在三项具有挑战性的灵巧操控任务中进行了验证实验，涵盖了多样化的条件，并对8名参与者进行了用户研究。 我们展示了所提出的方法在执行分支选择和异常检测方面对于初学者用户的可靠性，在576个真实机器人演示中分别达到了90.7%的分支选择准确率和87.9%的异常检测准确率。所有代码和数据均可用，以重现我们的实验成果（参见文中提供的网址）。
URL
https://arxiv.org/abs/2603.08057
