Abstract
Large Language Models (LLMs) continue to advance natural language processing with their ability to generate human-like text across a range of tasks. Despite the remarkable success of LLMs in Natural Language Processing (NLP), their performance in text summarization across various domains and datasets has not been comprehensively evaluated. At the same time, the ability to summarize text effectively without relying on extensive training data has become a crucial bottleneck. To address these issues, we present a systematic evaluation of six LLMs across four datasets: CNN/Daily Mail and NewsRoom (news), SAMSum (dialog), and ArXiv (scientific). By leveraging prompt engineering techniques including zero-shot and in-context learning, our study evaluates the performance using the ROUGE and BERTScore metrics. In addition, a detailed analysis of inference times is conducted to better understand the trade-off between summarization quality and computational efficiency. For Long documents, introduce a sentence-based chunking strategy that enables LLMs with shorter context windows to summarize extended inputs in multiple stages. The findings reveal that while LLMs perform competitively on news and dialog tasks, their performance on long scientific documents improves significantly when aided by chunking strategies. In addition, notable performance variations were observed based on model parameters, dataset properties, and prompt design. These results offer actionable insights into how different LLMs behave across task types, contributing to ongoing research in efficient, instruction-based NLP systems.
Abstract (translated)
大型语言模型（LLMs）通过其生成类似人类文本的能力，在自然语言处理方面取得了持续的进步，这一能力跨越了各种任务。尽管LLMs在自然语言处理（NLP）领域取得了非凡的成功，但它们在不同领域和数据集中的文本摘要性能尚未得到全面评估。同时，有效总结文本而不依赖于大量训练数据的能力已经成为一个关键瓶颈。为了解决这些问题，我们提出了对六种大型语言模型在四个数据集上的系统性评价：CNN/Daily Mail和NewsRoom（新闻），SAMSum（对话）以及ArXiv（科学）。通过利用包括零样本学习和上下文学习在内的提示工程技术，我们的研究使用ROUGE和BERTScore指标评估了这些模型的性能。此外，还进行了详细的推理时间分析，以便更好地理解摘要质量和计算效率之间的权衡。 针对长文档，我们引入了一种基于句子分割的战略，这使得具有较短上下文窗口的LLMs能够通过多阶段处理来总结较长输入内容。研究结果表明，在新闻和对话任务中，LLMs表现出了竞争性，而在科学文献摘要上，当借助分块策略时，其性能显著提升。此外，根据模型参数、数据集属性以及提示设计的不同，观察到了明显的性能差异。 这些成果为不同大型语言模型在各种任务类型中的行为提供了可操作的见解，并对高效的指令式自然语言处理系统的研究做出了贡献。
URL
https://arxiv.org/abs/2507.05123
