Abstract
Large language models primarily rely on inductive reasoning for decision making. This results in unreliable decisions when applied to real-world tasks that often present incomplete contexts and conditions. Thus, accurate probability estimation and appropriate interpretations are required to enhance decision-making reliability. In this paper, we propose a Bayesian inference framework called BIRD for large language models. BIRD provides controllable and interpretable probability estimation for model decisions, based on abductive factors, LLM entailment, as well as learnable deductive Bayesian modeling. Experiments show that BIRD produces probability estimations that align with human judgments over 65% of the time using open-sourced Llama models, outperforming the state-of-the-art GPT-4 by 35%. We also show that BIRD can be directly used for trustworthy decision making on many real-world applications.
Abstract (translated)
大语言模型主要依赖归纳推理来进行决策。这导致在应用于现实任务时，由于通常呈现不完整的语境和条件，因此不可靠的决策结果。因此，准确的概率估计和适当的解释是提高决策可靠性的必要条件。在本文中，我们提出了一个名为BIRD的大语言模型推理框架。BIRD为大型语言模型提供了可控制和可解释的概率估计，基于类推因素、LLM包含以及可学习演绎贝叶斯建模。实验结果表明，使用开源的Llama模型，BIRD在65%的时间里概率估计与人类判断相一致，比最先进的GPT-4模型快35%。我们还证明了BIRD可以直接用于许多现实应用的可信决策。
URL
https://arxiv.org/abs/2404.12494
