Abstract
Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VeriFastScore, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives ~4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VeriFastScore model demonstrates strong correlation with the original VeriScore pipeline at both the example level (r=0.80) and system level (r=0.94) while achieving an overall speedup of 6.6x (9.9x excluding evidence retrieval) over VeriScore. To facilitate future factuality research, we publicly release our VeriFastScore model and synthetic datasets.
Abstract (translated)
评估长文事实性的指标，如FactScore和VeriScore，通过将输入响应分解为原子声明，并逐一验证每个声明来工作。尽管这些方法有效且易于理解，但它们需要进行大量的大型语言模型（LLM）调用，并且可能需要长达100秒才能评估单个响应，这在大规模评估和训练场景中是不切实际的。为了应对这一挑战，我们提出了VeriFastScore，该方法利用合成数据对Llama3.1 8B进行微调，使其能够同时从给定文本中提取并根据Google搜索提供的证据验证所有可核实声明。 我们展示了这样一个任务不能通过使用封闭式LLM的少量提示来解决，因为其复杂性：模型需要接收平均约4K令牌的证据，并且需要同时分解声明、判断它们的可证实性和将它们与噪声证据进行对比。然而，我们的微调VeriFastScore模型在示例级别（r=0.80）和系统级别（r=0.94）上都显示出与原始VeriScore管道具有很强的相关性，并且比VeriScore整体速度提高了6.6倍（不包括证据检索则为9.9倍）。 为了促进未来的事实性研究，我们公开发布了我们的VeriFastScore模型和合成数据集。
URL
https://arxiv.org/abs/2505.16973
