Abstract
Large Language Models (LLMs) have recently achieved remarkable performance in long-context understanding. However, current long-context LLM benchmarks are limited by rigid context length, labor-intensive annotation, and the pressing challenge of label leakage issues during LLM training. Therefore, we propose \textsc{AcademicEval}, a live benchmark for evaluating LLMs over long-context generation tasks. \textsc{AcademicEval} adopts papers on arXiv to introduce several academic writing tasks with long-context inputs, \textit{i.e.}, \textsc{Title}, \textsc{Abstract}, \textsc{Introduction}, and \textsc{Related Work}, which cover a wide range of abstraction levels and require no manual labeling. Moreover, \textsc{AcademicEval} integrates high-quality and expert-curated few-shot demonstrations from a collected co-author graph to enable flexible context length. Especially, \textsc{AcademicEval} features an efficient live evaluation, ensuring no label leakage. We conduct a holistic evaluation on \textsc{AcademicEval}, and the results illustrate that LLMs perform poorly on tasks with hierarchical abstraction levels and tend to struggle with long few-shot demonstrations, highlighting the challenge of our benchmark. Through experimental analysis, we also reveal some insights for enhancing LLMs' long-context modeling capabilities. Code is available at this https URL
Abstract (translated)
最近，大型语言模型（LLMs）在处理长上下文理解方面取得了显著的性能提升。然而，现有的长上下文评估基准面临固定上下文长度、标注工作量大以及训练过程中标签泄露等问题。为此，我们提出了一个名为\textsc{AcademicEval} 的实时基准测试工具，用于评测LLM在长上下文生成任务中的表现。 \textsc{AcademicEval} 使用arXiv上的论文引入了几项涉及长上下文输入的学术写作任务，即标题、摘要、引言和相关工作部分。这些任务涵盖了广泛的抽象层级，并且无需人工标注。此外，\textsc{AcademicEval} 结合了一种高质量的、由专家策划的少量样本演示方法，该方法基于收集到的共作者图来实现灵活的上下文长度。特别地，\textsc{AcademicEval} 设计了高效的实时评估机制，确保没有标签泄露问题。 我们对\textsc{AcademicEval} 进行了全面评测，并且结果表明LLM在具有层次抽象水平的任务上表现不佳，而且难以处理长的少量样本演示，这突显出了该基准测试的独特挑战。通过实验分析，我们也揭示了一些增强LLM长上下文建模能力的见解。 代码可在以下链接获取：[提供具体的URL地址]
URL
https://arxiv.org/abs/2510.17725
