Abstract
Task arithmetic has emerged as a simple yet powerful technique for model merging, enabling the combination of multiple finetuned models into one. Despite its empirical success, a clear theoretical explanation of why and when it works is lacking. This paper provides a rigorous theoretical foundation for task arithmetic by establishing a connection between task vectors and gradients of the task losses. We show that under standard gradient descent, a task vector generated from one epoch of finetuning is exactly equivalent to the negative gradient of the loss, scaled by the learning rate. For the practical multi-epoch setting, we prove that this equivalence holds approximately, with a second-order error term that we explicitly bound for feed-forward networks. Our empirical analysis across seven vision benchmarks corroborates our theory, demonstrating that the first-epoch gradient dominates the finetuning trajectory in both norm and direction. A key implication is that merging models finetuned for only a single epoch often yields performance comparable to merging fully converged models. These findings reframe task arithmetic as a form of approximate multitask learning, providing a clear rationale for its effectiveness and highlighting the critical role of early training dynamics in model merging.
Abstract (translated)
任务算术作为一种简单而强大的模型合并技术已经浮现，它能够将多个微调后的模型合成为一个。尽管在实践中取得了成功，但关于该方法为何以及何时有效的确切理论解释却一直缺失。本文为任务算术提供了一个严格的理论基础，通过建立任务向量与任务损失梯度之间的联系来实现这一点。我们展示了，在标准的梯度下降过程中，从一个微调周期生成的任务向量等同于负梯度乘以学习率。对于实际中的多轮次设置，我们证明这种等价性近似成立，并且对前馈网络中的二阶误差项进行了明确界定。 我们的实证分析跨越了七个视觉基准测试集，验证了这一理论，表明第一个微调周期的梯度在范数和方向上主导着后续的微调过程。一个关键结论是，合并仅经过单个周期微调的模型通常可以取得与完全收敛后的模型相当的性能。这些发现将任务算术重新定义为一种近似的多任务学习形式，并清晰地解释了其有效性，同时也强调了早期训练动态在模型合并中的关键作用。
URL
https://arxiv.org/abs/2508.16082
