Abstract
Uncertainty quantification (UQ) is a prominent approach for eliciting truthful answers from large language models (LLMs). To date, information-based and consistency-based UQ have been the dominant UQ methods for text generation via LLMs. Density-based methods, despite being very effective for UQ in text classification with encoder-based models, have not been very successful with generative LLMs. In this work, we adapt Mahalanobis Distance (MD) - a well-established UQ technique in classification tasks - for text generation and introduce a new supervised UQ method. Our method extracts token embeddings from multiple layers of LLMs, computes MD scores for each token, and uses linear regression trained on these features to provide robust uncertainty scores. Through extensive experiments on eleven datasets, we demonstrate that our approach substantially improves over existing UQ methods, providing accurate and computationally efficient uncertainty scores for both sequence-level selective generation and claim-level fact-checking tasks. Our method also exhibits strong generalization to out-of-domain data, making it suitable for a wide range of LLM-based applications.
Abstract (translated)
不确定性量化（UQ）是一种从大型语言模型（LLM）中获取真实答案的重要方法。迄今为止，基于信息和一致性的方法是通过LLM进行文本生成的主要不确定性量化方法。尽管密度基方法在基于编码器的模型的文本分类任务中的不确定性量化方面非常有效，但它们在生成式LLM上并不成功。在这项工作中，我们改编了Mahalanobis距离（MD）——一种在分类任务中广泛应用的UQ技术，并引入了一种新的监督UQ方法。我们的方法从LLM的不同层提取token嵌入，为每个token计算MD分数，并使用基于这些特征训练的线性回归模型提供稳健的不确定性评分。通过在十一项数据集上的广泛实验，我们展示了该方法比现有的UQ方法有显著改进，能够为序列级的选择性生成和声明级别的事实核查任务提供准确且计算效率高的不确定性评分。此外，我们的方法还表现出强大的跨域泛化能力，适用于广泛的基于LLM的应用场景。
URL
https://arxiv.org/abs/2502.14427
