Abstract
Large Language Models (LLMs) often suffer from hallucinations: output content that is not grounded in the input context, when performing long-form text generation tasks such as summarization. Prior works have shown that hallucinations can be reduced by iteratively critiquing and refining previously generated outputs using either the same model or a more powerful teacher model as the critique. However, these approaches either require additional test-time compute or assume access to more powerful teacher models, making them costly and less practical. In this work, we propose Self Critique and Refinement-based Preference Optimization (SCRPO), which is a self-supervised training framework that first constructs a preference dataset by leveraging the LLM's own critique and refinement capabilities, and then applies preference learning to improve the same LLM for faithful summarization. Experiments on three summarization benchmarks (XSUM CNNDM and SAMSum), demonstrate that our approach outperforms state-of-the-art self-supervised learning methods in terms of faithfulness metrics while either maintaining or improving other metrics that measure the overall quality of the summary. Moreover, compared to test-time refinement, our approach not only improves efficiency but also results in more faithful summaries.
Abstract (translated)
大型语言模型（LLM）在执行长文本生成任务如摘要时，常常会出现幻觉现象：输出的内容与输入上下文无关。先前的研究表明，通过迭代地使用相同的模型或更强大的教师模型对之前生成的输出进行批判和细化，可以减少这些幻觉现象。然而，这种方法要么需要额外的测试计算资源，要么假设能够访问到更强大的教师模型，这使得它们既昂贵又不太实用。 在本研究中，我们提出了基于自我批评与优化偏好的改进方法（SCRPO），这是一种自监督训练框架。该框架首先通过利用LLM自身的批判和细化能力来构建偏好数据集，然后应用偏好学习以提高同一LLM的忠实摘要生成性能。在三个摘要基准测试（XSUM、CNNDM 和 SAMSum）上的实验表明，我们提出的方法在忠实度指标方面优于当前最先进的自监督学习方法，并且还能保持或提升衡量摘要总体质量的其他指标。 此外，与测试时间细化相比，我们的方法不仅提高了效率，还生成了更加忠实的摘要。
URL
https://arxiv.org/abs/2512.05387
