Abstract
As Large Language Models (LLMs) have become more advanced, they have outpaced our abilities to accurately evaluate their quality. Not only is finding data to adequately probe particular model properties difficult, but evaluating the correctness of a model's freeform generation alone is a challenge. To address this, many evaluations now rely on using LLMs themselves as judges to score the quality of outputs from other LLMs. Evaluations most commonly use a single large model like GPT4. While this method has grown in popularity, it is costly, has been shown to introduce intramodel bias, and in this work, we find that very large models are often unnecessary. We propose instead to evaluate models using a Panel of LLm evaluators (PoLL). Across three distinct judge settings and spanning six different datasets, we find that using a PoLL composed of a larger number of smaller models outperforms a single large judge, exhibits less intra-model bias due to its composition of disjoint model families, and does so while being over seven times less expensive.
Abstract (translated)
随着大型语言模型（LLMs）变得更加先进，它们已经超越了我们对准确评估其质量的能力。不仅找到足够的数据来充分检验特定模型的特性困难，而且仅评估模型的自由形式生成正确性就是一个挑战。为了解决这个问题，许多评估现在依赖使用LLMs本身作为评判者来评分其他LLMs的输出质量。最常见的评估方法使用单个大型模型GPT4。虽然这种方法变得越来越受欢迎，但代价昂贵，还表明引入了内模型偏见。在这项工作中，我们发现，大型模型往往是不必要的。我们提出了一种通过一组LLM评估者（PoLL）来评估模型的方法。在三个不同的评判设置和跨越六个不同数据集的情况下，我们发现，由多个较小的模型组成的PoLL比单个大型评判者表现更好，因为它由不同的模型家族组成，表现出了较少的内模型偏见。而且，即使花费的成本是单个大型评判者的七倍之多，它也表现出更好的性能。
URL
https://arxiv.org/abs/2404.18796
