Abstract
Estimating uncertainty for AI agents in real-world multi-turn tool-using interaction with humans is difficult because failures are often triggered by sparse critical episodes (e.g., looping, incoherent tool use, or user-agent miscoordination) even when local generation appears confident. Existing uncertainty proxies focus on single-shot text generation and therefore miss these trajectory-level breakdown signals. We introduce TRACER, a trajectory-level uncertainty metric for dual-control Tool-Agent-User interaction. TRACER combines content-aware surprisal with situational-awareness signals, semantic and lexical repetition, and tool-grounded coherence gaps, and aggregates them using a tail-focused risk functional with a MAX-composite step risk to surface decisive anomalies. We evaluate TRACER on $\tau^2$-bench by predicting task failure and selective task execution. To this end, TRACER improves AUROC by up to 37.1% and AUARC by up to 55% over baselines, enabling earlier and more accurate detection of uncertainty in complex conversational tool-use settings. Our code and benchmark are available at this https URL.
Abstract (translated)
在现实世界中，AI代理与人类进行多轮工具使用交互时，估算不确定性非常困难。因为即使局部生成看起来很自信，失败也可能由稀疏的关键事件（如循环、不连贯的工具使用或用户-代理协调失误）触发。现有的不确定性的替代指标主要关注单次文本生成，因此忽略了这些轨迹层面崩溃信号。 我们引入了TRACER，这是一种针对双控制工具-代理-用户的交互的轨迹级不确定性度量方法。TRACER结合了基于内容的意外性（surprisal）、情境感知信号、语义和词汇重复以及工具相关的连贯性差距，并使用一种尾部聚焦的风险函数来汇总这些因素，同时采用MAX复合步骤风险以揭示决定性的异常情况。 我们在$\tau^2$-bench上评估了TRACER，通过预测任务失败和选择性任务执行的能力。为此，与基线相比，TRACER将AUROC提高了最多37.1%，将AUARC提高了最多55%，从而在复杂的对话工具使用环境中实现了更早且更准确的不确定性检测。 我们的代码和基准测试可在以下网址获取：[this https URL]。
URL
https://arxiv.org/abs/2602.11409
