Abstract
The rapid development of large language models (LLMs) has shown promising practical results. However, their low interpretability often leads to errors in unforeseen circumstances, limiting their utility. Many works have focused on creating comprehensive evaluation systems, but previous benchmarks have primarily assessed problem-solving abilities while neglecting the response's uncertainty, which may result in unreliability. Recent methods for measuring LLM reliability are resource-intensive and unable to test black-box models. To address this, we propose UBENCH, a comprehensive benchmark for evaluating LLM reliability. UBENCH includes 3,978 multiple-choice questions covering knowledge, language, understanding, and reasoning abilities. Experimental results show that UBENCH has achieved state-of-the-art performance, while its single-sampling method significantly saves computational resources compared to baseline methods that require multiple samplings. Additionally, based on UBENCH, we evaluate the reliability of 15 popular LLMs, finding GLM4 to be the most outstanding, closely followed by GPT-4. We also explore the impact of Chain-of-Thought prompts, role-playing prompts, option order, and temperature on LLM reliability, analyzing the varying effects on different LLMs.
Abstract (translated)
大规模语言模型（LLMs）的快速发展已经取得了有前景的实践成果。然而，它们往往具有较低的可解释性，这导致在意外情况下出现错误，限制了它们的实用性。许多工作集中在创建全面评估系统，但之前的基准主要评估了解决问题的能力，而忽视了响应的不确定性，这可能导致不可靠。最近用于衡量LLM可靠性的方法资源密集且无法测试黑盒模型。为解决这个问题，我们提出了UBENCH，一个全面评估LLM可靠性的基准。UBENCH包括3978个多选题，覆盖知识、语言、理解和推理能力。实验结果表明，UBENCH已经达到了最先进的水平，而其单采样方法在基线方法需要进行多次采样时显著节省了计算资源。此外，根据UBENCH，我们评估了15个流行LLM的可靠性，发现GLM4是最出色的，紧随其后的是GPT-4。我们还研究了思考链提示、角色扮演提示、选项顺序和温度对LLM可靠性的影响，分析不同LLM之间这些因素的变化效果。
URL
https://arxiv.org/abs/2406.12784
