Abstract
Aligned Large Language Models (LLMs) showcase remarkable versatility, capable of handling diverse real-world tasks. Meanwhile, aligned LLMs are also expected to exhibit speciality, excelling in specific applications. However, fine-tuning with extra data, a common practice to gain speciality, often leads to catastrophic forgetting (CF) of previously acquired versatility, hindering the model's performance across diverse tasks. In response to this challenge, we propose CoFiTune, a coarse to fine framework in an attempt to strike the balance between speciality and versatility. At the coarse-grained level, an empirical tree-search algorithm is utilized to pinpoint and update specific modules that are crucial for speciality, while keeping other parameters frozen; at the fine-grained level, a soft-masking mechanism regulates the update to the LLMs, mitigating the CF issue without harming speciality. In an overall evaluation of both speciality and versatility, CoFiTune consistently outperforms baseline methods across diverse tasks and model scales. Compared to the full-parameter SFT, CoFiTune leads to about 14% versatility improvement and marginal speciality loss on a 13B model. Lastly, based on further analysis, we provide a speculative insight into the information forwarding process in LLMs, which helps explain the effectiveness of the proposed method. The code is available at this https URL.
Abstract (translated)
aligned large language models (LLMs) 展示了令人印象深刻的多才多艺，能够处理多样的人工现实任务。同时，与对齐的 LLMs 也预计将表现出专业性，在特定应用中表现出色。然而，通过额外的数据进行微调，这是一种常见的获得专业性的方法，往往会导致对先前获得的多样性的灾难性遗忘（CF），从而阻碍模型在各种任务上的表现。为了应对这个挑战，我们提出了 CoFiTune，一种在专业性和多样性之间取得平衡的尝试。在粗粒度级别，采用了一种经验性的树搜索算法来确定和更新对于专业性至关重要的一些模块，而其他参数则保持不变；在细粒度级别，采用了一种软掩码机制来调节对 LLMs 的更新，从而减轻 CF 问题，同时不损害专业性。在多样任务和模型规模的综合评估中，CoFiTune 始终在基线方法上表现优异。与完整参数 SFT 相比，CoFiTune导致约 14% 的多样性改进和约 13B 模型的边缘专业性损失。最后，根据进一步的分析，我们提供了一个关于 LLMs 中信息传递过程的启示性的见解，这有助于解释所提出方法的有效性。代码可在此链接下载：https://www.aclweb.org/anthology/N22-2161
URL
https://arxiv.org/abs/2404.10306
