Abstract
As Large Language Model (LLM) agents increasingly operate in complex environments with real-world consequences, their safety becomes critical. While uncertainty quantification is well-studied for single-turn tasks, multi-turn agentic scenarios with real-world tool access present unique challenges where uncertainties and ambiguities compound, leading to severe or catastrophic risks beyond traditional text generation failures. We propose using "quitting" as a simple yet effective behavioral mechanism for LLM agents to recognize and withdraw from situations where they lack confidence. Leveraging the ToolEmu framework, we conduct a systematic evaluation of quitting behavior across 12 state-of-the-art LLMs. Our results demonstrate a highly favorable safety-helpfulness trade-off: agents prompted to quit with explicit instructions improve safety by an average of +0.39 on a 0-3 scale across all models (+0.64 for proprietary models), while maintaining a negligible average decrease of -0.03 in helpfulness. Our analysis demonstrates that simply adding explicit quit instructions proves to be a highly effective safety mechanism that can immediately be deployed in existing agent systems, and establishes quitting as an effective first-line defense mechanism for autonomous agents in high-stakes applications.
Abstract (translated)
随着大型语言模型（LLM）代理越来越多地在具有现实世界后果的复杂环境中运行，其安全性变得至关重要。虽然不确定性量化对于单轮任务已有深入研究，但多轮代理场景中与现实世界工具互动的情况带来了独特的挑战：不确定性和模糊性会累积起来，导致超出传统文本生成失败的风险，可能会引发严重或灾难性的风险。我们提出使用“退出”作为简单而有效的行为机制，使LLM代理在缺乏信心的情况下能够识别并从当前情境中撤出。 利用ToolEmu框架，我们在12种最先进的LLM上对退出行为进行了系统的评估。我们的结果显示，在安全性和有用性之间存在一个高度有利的权衡：通过明确指令提示代理退出，在0到3的评分尺度上，所有模型的安全性能平均提高+0.39（专有模型为+0.64），同时保持了微不足道的帮助性能平均下降-0.03。我们的分析表明，简单地添加明确的退出指令被证明是一种高度有效的安全机制，可以立即部署在现有的代理系统中，并且确定“退出”作为自主代理在高风险应用中的有效一线防御机制。
URL
https://arxiv.org/abs/2510.16492
