Abstract
Large language models (LLMs) with one or more fine-tuning phases have become a necessary step to unlock various capabilities, enabling LLMs to follow natural language instructions or align with human preferences. However, it carries the risk of catastrophic forgetting during sequential training, the parametric knowledge or the ability learned in previous stages may be overwhelmed by incoming training data. In this paper, we find that by regularly resetting partial parameters, LLMs can restore some of the original knowledge. Inspired by this, we introduce Half Fine-Tuning (HFT) for LLMs, as a substitute for full fine-tuning (FFT), to mitigate the forgetting issues, where half of the parameters are selected to learn new tasks while the other half are frozen to remain previous knowledge. We provide a feasibility analysis from the perspective of optimization and interpret the parameter selection operation as a regularization term. Without changing the model architecture, HFT could be seamlessly integrated into existing fine-tuning frameworks. Extensive experiments and analysis on supervised fine-tuning, direct preference optimization, and continual learning consistently demonstrate the effectiveness, robustness, and efficiency of HFT. Compared with FFT, HFT not only significantly alleviates the forgetting problem, but also achieves the best performance in a series of downstream benchmarks, with an approximately 30% reduction in training time.
Abstract (translated)
大型语言模型（LLMs）通过一次或多次微调阶段已成为解锁各种功能的有必要的一步，使LLM能够遵循自然语言指令或与人类偏好对齐。然而，在序列训练过程中，它们可能面临灾难性遗忘的风险，以前阶段学到的参数或能力可能会被传入的训练数据所压倒。在本文中，我们发现，通过定期重置部分参数，LLM可以恢复一些原始知识。受到这一启发，我们引入了半微调（HFT）用于LLM，作为完全微调（FFT）的替代方案，以减轻遗忘问题，其中一半参数用于学习新任务，而另一半参数则保持不变以保留先前的知识。我们从优化的角度进行了可行性分析，并将参数选择操作解释为正则化项。在没有改变模型架构的情况下，HFT可以轻松地融入现有的微调框架。对监督微调、直接偏好优化和持续学习的大规模实验和分析都证实了HFT的有效性、稳健性和效率。与FFT相比，HFT不仅显著减轻了遗忘问题，而且在一系列下游基准测试中实现了最佳性能，训练时间约减少30%。
URL
https://arxiv.org/abs/2404.18466
