Abstract
The increasing threat of disinformation calls for automating parts of the fact-checking pipeline. Identifying text segments requiring fact-checking is known as claim detection (CD) and claim check-worthiness detection (CW), the latter incorporating complex domain-specific criteria of worthiness and often framed as a ranking task. Zero- and few-shot LLM prompting is an attractive option for both tasks, as it bypasses the need for labeled datasets and allows verbalized claim and worthiness criteria to be directly used for prompting. We evaluate the LLMs' predictive and calibration accuracy on five CD/CW datasets from diverse domains, each utilizing a different worthiness criterion. We investigate two key aspects: (1) how best to distill factuality and worthiness criteria into a prompt and (2) what amount of context to provide for each claim. To this end, we experiment with varying the level of prompt verbosity and the amount of contextual information provided to the model. Our results show that optimal prompt verbosity is domain-dependent, adding context does not improve performance, and confidence scores can be directly used to produce reliable check-worthiness rankings.
Abstract (translated)
随着虚假信息的威胁越来越大，需要自动化事实核查管道中的某些部分。识别需要核实的事实文本段落称为断言检测（CD），后者包括复杂的领域特定价值标准，通常被框架为一个排名任务。零和少样本LLM提示对于两种任务来说都是具有吸引力的选择，因为它绕过了需要标记数据集的需求，并允许直接使用口头断言和价值标准进行提示。我们评估了五种不同领域的CD/CW数据集上LLM的预测和校准准确性，每种数据集都使用不同的价值标准。我们研究了两个关键方面：（1）如何将事实性和价值标准精炼成提示；（2）为每个断言提供多少上下文。为此，我们尝试改变提示的措辞和提供给模型的上下文信息水平。我们的结果表明，最优的提示措辞是受领域影响的，增加上下文并不能提高性能，而自信分数可以直接用于产生可靠的检查价值排名。
URL
https://arxiv.org/abs/2404.12174
