Abstract
Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. While LLMs have improved at processing long inputs, MD contexts still present challenges, such as managing inter-document dependencies, redundancy, and incoherent structures. We introduce MDCure, a scalable and effective fine-tuning pipeline to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human annotated data. MDCure is based on generation of high-quality synthetic MD instruction data from sets of related articles via targeted prompts. We further introduce MDCureRM, a multi-objective reward model which filters generated data based on their training utility for MD settings. With MDCure, we fine-tune a variety of LLMs, from the FlanT5, Qwen2, and LLAMA3.1 model families, up to 70B parameters in size. Extensive evaluations on a wide range of MD and long-context benchmarks spanning various tasks show MDCure consistently improves performance over pre-trained baselines and over corresponding base models by up to 75.5%. Our code, datasets, and models are available at this https URL.
Abstract (translated)
多文档（MD）处理对于大型语言模型（LLMs）在总结和问答等实际任务中处理大量文档集至关重要。虽然LLMs在处理长输入方面已经有所改进，但多文档上下文仍然存在挑战，如管理文档间的依赖关系、冗余和不一致的结构。我们介绍了一种可扩展且有效的微调管道MDCure，用于增强LLMs的多文档能力，而无需预训练的计算成本或依赖人工标注的数据。MDCure基于通过定向提示从相关文章集生成高质量的合成MD指令数据。此外，我们还介绍了MDCureRM，这是一种多目标奖励模型，它根据生成数据在多文档设置中的训练效用对其进行过滤。使用MDCure，我们可以微调来自FlanT5、Qwen2和LLAMA3.1等多个模型家族的不同规模的LLMs，参数量可达70B。广泛的评估涵盖了各种MD和长上下文基准测试以及不同任务，结果表明MDCure在预训练基线和相应基础模型上的性能持续提升了高达75.5%。我们的代码、数据集和模型可以在以下链接中找到：[此HTTPS URL]。
URL
https://arxiv.org/abs/2410.23463
