Abstract
While vision-language-action models (VLAs) have shown promising robotic behaviors across a diverse set of manipulation tasks, they achieve limited success rates when deployed on novel tasks out-of-the-box. To allow these policies to safely interact with their environments, we need a failure detector that gives a timely alert such that the robot can stop, backtrack, or ask for help. However, existing failure detectors are trained and tested only on one or a few specific tasks, while VLAs require the detector to generalize and detect failures also in unseen tasks and novel environments. In this paper, we introduce the multitask failure detection problem and propose SAFE, a failure detector for generalist robot policies such as VLAs. We analyze the VLA feature space and find that VLAs have sufficient high-level knowledge about task success and failure, which is generic across different tasks. Based on this insight, we design SAFE to learn from VLA internal features and predict a single scalar indicating the likelihood of task failure. SAFE is trained on both successful and failed rollouts, and is evaluated on unseen tasks. SAFE is compatible with different policy architectures. We test it on OpenVLA, $\pi_0$, and $\pi_0$-FAST in both simulated and real-world environments extensively. We compare SAFE with diverse baselines and show that SAFE achieves state-of-the-art failure detection performance and the best trade-off between accuracy and detection time using conformal prediction. More qualitative results can be found at this https URL.
Abstract (translated)
尽管视觉语言行动模型（VLAs）在各种操作任务中展示了有前景的机器人行为，但当部署到全新的任务时，它们的成功率有限。为了使这些策略能够安全地与其环境互动，我们需要一个故障检测器，能够在关键时刻发出警报，让机器人可以停止、回溯或请求帮助。然而，现有的故障检测器仅在特定的一个或几个任务上进行训练和测试，而VLAs则需要检测器能在未见过的任务和新环境中泛化并识别故障。 在这篇论文中，我们引入了多任务故障检测问题，并提出了SAFE——一个为包括VLAs在内的通才机器人策略设计的故障检测器。通过对VLA特征空间的分析，我们发现VLAs对任务的成功与失败拥有足够的高层次知识，这种知识在不同任务间具有通用性。基于这一洞察，我们将SAFE设计成能够从VLA内部特性学习，并预测一个单一标量值来表示任务失败的可能性。SAFE是在成功和失败的情况下进行训练的，且评估时使用的是未见过的任务。此外，SAFE与不同的策略架构兼容。 我们在模拟环境和现实世界中对OpenVLA、$\pi_0$以及$\pi_0$-FAST进行了广泛的测试。我们将SAFE与各种基线进行了比较，并展示了它在故障检测性能上取得了最先进的成果，且使用一致预测实现了准确性与检测时间的最佳平衡。更多定性结果可以在[该链接](https://thisisnotalink.com)中找到。 请注意，提供的URL实际为示例链接，请根据实际情况替换为正确的网址。
URL
https://arxiv.org/abs/2506.09937
