Abstract
With more open-source models available for diverse tasks, model merging has gained attention by combining models into one, reducing training, storage, and inference costs. Current research mainly focuses on model merging for full fine-tuning, overlooking the popular LoRA. However, our empirical analysis reveals that: a) existing merging methods designed for full fine-tuning perform poorly on LoRA; b) LoRA modules show much larger parameter magnitude variance than full fine-tuned weights; c) greater parameter magnitude variance correlates with worse merging performance. Considering that large magnitude variances cause deviations in the distribution of the merged parameters, resulting in information loss and performance degradation, we propose a Decoupled and Orthogonal merging approach(DO-Merging). By separating parameters into magnitude and direction components and merging them independently, we reduce the impact of magnitude differences on the directional alignment of the merged models, thereby preserving task information. Furthermore, we introduce a data-free, layer-wise gradient descent method with orthogonal constraints to mitigate interference during the merging of direction components. We provide theoretical guarantees for both the decoupling and orthogonal components. And we validate through extensive experiments across vision, language, and multi-modal domains that our proposed DO-Merging can achieve significantly higher performance than existing merging methods at a minimal cost. Notably, each component can be flexibly integrated with existing methods, offering near free-lunch improvements across tasks.
Abstract (translated)
随着开源模型的增多，能够为各种任务服务，模型合并（将多个模型整合成一个）越来越受到关注。通过这种方法可以减少训练、存储和推理的成本。然而，当前的研究主要集中在全面微调（full fine-tuning）上的模型合并上，而忽略了流行的LoRA方法。我们的实证分析表明： a) 为全面微调设计的现有合并方法在处理LoRA时表现不佳； b) LoRA模块显示出比完全微调权重更大的参数幅度变化； c) 更大的参数幅度变化与较差的合并性能相关联。 考虑到大范围的幅度差异会导致合并参数分布中的偏差，从而导致信息丢失和性能下降，我们提出了一种解耦正交合并方法（Decoupled and Orthogonal merging, DO-Merging）。通过将参数分解为幅度和方向分量，并独立地对它们进行合并，我们可以减少幅度差异对合并模型方向一致性的负面影响，从而保留任务相关信息。 此外，为了在方向分量的合并过程中减轻干扰，我们引入了一种无数据依赖、逐层梯度下降的方法，该方法带有正交约束。我们为解耦和正交部分都提供了理论保证。通过广泛的实验验证（涵盖视觉、语言以及跨模态领域），我们的DO-Merging方法可以在最低成本下实现比现有合并方法显著更高的性能。 值得注意的是，每个组件都可以灵活地与现有的方法集成，从而在各种任务中提供近乎无代价的改进效果。
URL
https://arxiv.org/abs/2505.15875
