Abstract
The multi-task learning ($MTL$) paradigm aims to simultaneously learn multiple tasks within a single model capturing higher-level, more general hidden patterns that are shared by the tasks. In deep learning, a significant challenge in the backpropagation training process is the design of advanced optimisers to improve the convergence speed and stability of the gradient descent learning rule. In particular, in multi-task deep learning ($MTDL$) the multitude of tasks may generate potentially conflicting gradients that would hinder the concurrent convergence of the diverse loss functions. This challenge arises when the gradients of the task objectives have either different magnitudes or opposite directions, causing one or a few to dominate or to interfere with each other, thus degrading the training process. Gradient surgery methods address the problem explicitly dealing with conflicting gradients by adjusting the overall gradient trajectory. This work introduces a novel gradient surgery method, the Similarity-Aware Momentum Gradient Surgery (SAM-GS), which provides an effective and scalable approach based on a gradient magnitude similarity measure to guide the optimisation process. The SAM-GS surgery adopts gradient equalisation and modulation of the first-order momentum. A series of experimental tests have shown the effectiveness of SAM-GS on synthetic problems and $MTL$ benchmarks. Gradient magnitude similarity plays a crucial role in regularising gradient aggregation in $MTDL$ for the optimisation of the learning process.
Abstract (translated)
多任务学习（MTL）范式旨在通过单个模型同时学习多个任务，捕捉这些任务共有的更高层次、更具普遍性的隐含模式。在深度学习领域，反向传播训练过程中的一大挑战是设计高级优化器以提升梯度下降学习规则的收敛速度和稳定性。特别是在多任务深度学习（MTDL）中，众多的任务可能会产生潜在冲突的梯度，阻碍各个多样化的损失函数的同时收敛。这一挑战源于任务目标的梯度在幅度或方向上的差异，导致某些梯度占据主导地位或是相互干扰，从而破坏训练过程。梯度手术方法通过调整整体梯度轨迹来明确解决冲突梯度的问题。 本文介绍了一种新的梯度手术方法——相似性感知动量梯度手术（SAM-GS），它基于梯度幅度的相似性测量提供了一个有效且可扩展的方式来引导优化过程。SAM-GS手术采用梯度均衡化和一阶动量调节的方式进行操作。一系列实验测试表明，SAM-GS在合成问题以及多任务学习基准上表现出了有效性。梯度幅度相似性在MTDL中对于正则化梯度聚合以优化学习过程起着关键作用。
URL
https://arxiv.org/abs/2506.06130
