Abstract
This paper develops an ensemble method for fine-tuning a language model to multiple datasets. Existing methods, such as quantized LoRA (QLoRA), are efficient when adapting to a single dataset. When training on multiple datasets of different tasks, a common setup in practice, it remains unclear how to design an efficient adaptation for fine-tuning language models. We propose to use an ensemble of multiple smaller adapters instead of a single adapter per task. We design an efficient algorithm that partitions $n$ datasets into $m$ groups, where $m$ is typically much smaller than $n$ in practice, and train one adapter for each group before taking a weighted combination to form the ensemble. The algorithm leverages a first-order approximation property of low-rank adaptation to quickly obtain the fine-tuning performances of dataset combinations since methods like LoRA stay close to the base model. Hence, we use the gradients of the base model to estimate its behavior during fine-tuning. Empirically, this approximation holds with less than $1\%$ error on models with up to $34$ billion parameters, leading to an estimation of true fine-tuning performances under $5\%$ error while speeding up computation compared to base fine-tuning by $105$ times. When applied to fine-tune Llama and GPT models on ten text classification tasks, our approach provides up to $10\%$ higher average test accuracy over QLoRA, with only $9\%$ more FLOPs. On a Llama model with $34$ billion parameters, an ensemble of QLoRA increases test accuracy by $3\%$ compared to QLoRA, with only $8\%$ more FLOPs.
Abstract (translated)
本文提出了一种针对多数据集微调语言模型的集成方法。现有技术如量化低秩适应（QLoRA）在单一数据集上的应用效率很高，但在面对不同任务的多个数据集进行训练时，如何设计高效的微调策略仍不清楚。我们建议使用多个较小适配器的集合而不是为每个任务单独创建一个适配器。具体来说，我们的方法将 $n$ 个数据集划分为 $m$ 组（其中 $m$ 在实践中通常远小于 $n$），然后为每一组训练一个适配器，并通过加权组合形成集成模型。 该算法利用了低秩适应的一阶近似性质，可以通过基础模型的梯度快速估计不同数据集组合在微调后的性能。这是因为像LoRA这样的方法在整个微调过程中仍保持接近基础模型的状态。因此，在计算上较为昂贵的实际微调之前，我们使用基础模型的梯度来估算其行为。 实验表明，这种近似误差低于1%，适用于高达340亿参数规模的语言模型，并且能够以约5%的误差范围内准确估计真正的微调性能，同时将基础模型微调的时间缩短了105倍。在对Llama和GPT模型进行十项文本分类任务的微调时，我们的方法相比QLoRA可提供高达10%的平均测试精度提升，并且仅增加了9%的浮点运算次数（FLOPs）。对于一个包含340亿参数的Llama模型而言，在使用QLoRA的基础上采用集合方法可以将测试准确度提高3%，同时只增加了8%的FLOPs。
URL
https://arxiv.org/abs/2505.21930
