Abstract
Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce outputs that are incomplete or selectively omit key information. In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies, including hallucinations. In this study, we address the challenge of evaluating the comprehensiveness of LLM-generated texts, focusing on the detection of missing information or underrepresented viewpoints. We investigate three automated evaluation strategies: (1) an NLI-based method that decomposes texts into atomic statements and uses natural language inference (NLI) to identify missing links, (2) a Q&A-based approach that extracts question-answer pairs and compares responses across sources, and (3) an end-to-end method that directly identifies missing content using LLMs. Our experiments demonstrate the surprising effectiveness of the simple end-to-end approach compared to more complex methods, though at the cost of reduced robustness, interpretability and result granularity. We further assess the comprehensiveness of responses from several popular open-weight LLMs when answering user queries based on multiple sources.
Abstract (translated)
尽管大型语言模型（LLM）在广泛的任务中表现出色，但它们也被发现经常会生成不完整或有选择性地省略关键信息的输出。在这种情况下，在敏感领域，这些遗漏可能会造成与事实错误相当的危害，包括幻觉现象。在这项研究中，我们解决了一个评估LLM生成文本全面性的挑战，重点在于检测缺失的信息或被边缘化的观点。我们调查了三种自动评估策略：（1）基于自然语言推理（NLI）的方法，将文本分解为原子陈述，并使用自然语言推理来识别缺失的联系；（2）基于问答的方式，提取问题-答案对并跨源比较响应；以及（3）一种端到端方法，直接利用LLM来识别缺少的内容。我们的实验表明，尽管简单端到端的方法在稳健性、可解释性和结果粒度方面有所牺牲，但它比复杂方法更有效地检测到了缺失信息。我们进一步评估了几个流行的开源LLM在回答基于多个来源的用户查询时响应的全面性。
URL
https://arxiv.org/abs/2510.07926
