Abstract
High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1\%, with no significant decline observed in other general capabilities.
Abstract (translated)
高质量的大规模指令对于调整大规模语言模型（LLM）至关重要，然而，在自然语言理解（NLU）领域存在严重的指令短缺。以前构建NLU指令的工作主要集中在信息抽取（IE）上，忽略了诸如机器阅读理解、问答和文本分类等任务。此外，数据缺乏多样性导致经过训练的LLMs在其他NLU任务中的泛化能力下降，并且基础模型的基本能力也明显减弱。为了解决这个问题，我们提出了Hum，这是一个针对NLU任务的大规模高质量合成指令语料库，旨在增强LLM的NLU能力。具体来说，Hum包括信息抽取（无论是封闭式还是开放式）、机器阅读理解、文本分类和指令通才任务，从而丰富了任务多样性。此外，我们引入了一种人与LLMs协作机制来生成指令，通过融入指南、偏好规则和格式变体，丰富了指令的多样性。我们在5个NLU任务和28个针对LLM的一般能力评估数据集上进行了广泛的实验。实验结果显示，Hum使六种LLM的NLU能力平均提升了3.1%，并且在其他一般能力方面没有显著下降。
URL
https://arxiv.org/abs/2502.03843
