Abstract
Rubric-based evaluation with large language models (LLMs) has become standard practice for assessing text generation at scale, yet the underlying techniques are scattered across papers with inconsistent terminology and partial solutions. We present a unified framework: each identified technique is paired with its realization in Autorubric, an open-source Python framework proposed in this paper. Autorubric supports binary, ordinal, and nominal criteria with configurable weights; single-judge and multi-judge ensemble evaluation with majority, weighted, unanimous, and any-vote aggregation; few-shot calibration with verdict-balanced sampling; and mitigations for position bias (option shuffling), verbosity bias (length penalties), and criterion conflation (per-criterion atomic evaluation with natural language explanations). The framework provides reliability metrics drawn from psychometrics (Cohen's $\kappa$, weighted $\kappa$, correlation coefficients, and distribution-level tests) alongside production infrastructure including response caching, checkpointing with resumable runs, multi-provider rate limiting, and cost tracking. We evaluate Autorubric on three benchmarks spanning educational assessment, deep research evaluation, and chatbot quality assessment, demonstrating that it produces results consistent with published benchmarks while exercising the framework's key capabilities: per-criterion binary evaluation with few-shot calibration (RiceChem), multi-judge ensemble evaluation across judge models (ResearcherBench), and mixed criterion types combining binary, ordinal, and nominal scales (CHARM-100). We also contribute CHARM-100, a 100-sample chatbot evaluation dataset with per-sample ground truth labels across all three criterion types, designed to stress-test rubric evaluation frameworks on heterogeneous criteria.
Abstract (translated)
基于评分标准的大规模语言模型（LLMs）评估已成为大规模文本生成评估的标准实践，但其底层技术分散在不同的论文中，并且术语不一致、解决方案也不完整。本文提出了一种统一的框架：每一种被识别的技术都与其在Autorubric中的实现相配对，Autorubric是本文提出的开源Python框架。该框架支持二元、序数和名义标准，并可以配置权重；单评价员和多评价员集成评估包括多数投票、加权投票、一致同意和任意投票聚合方法；少量样本校准采用判决平衡抽样；并且有缓解位置偏差（选项洗牌）、冗长偏见（长度惩罚）以及评判混淆的措施（每个评判点原子级评估并附带自然语言解释）。该框架提供了来自心理测量学的可靠性指标（Cohen's κ、加权κ、相关系数和分布水平测试），同时包括了生产基础设施，如响应缓存、可恢复运行的检查点、多提供商速率限制以及成本跟踪。我们通过三个涵盖教育评估、深度研究评估及聊天机器人质量评估的基准来评测Autorubric，证明它能产生与已发布基准一致的结果，同时也展示了其关键功能：基于少量样本校准的单评判标准二元评价（RiceChem），跨越不同评判模型的多评价员集成评价（ResearcherBench），以及结合了二元、序数和名义尺度的混合评判类型评估（CHARM-100）。我们还贡献了一个名为CHARM-100的数据集，它包含了100个样本的聊天机器人评估数据，并为所有三种评判标准提供了每个样本的真实标签，旨在考验评分标准评价框架在异构准则上的性能。
URL
https://arxiv.org/abs/2603.00077
