Abstract
Parameter-efficient fine-tuning methods have emerged as a promising solution for adapting pre-trained models to various downstream tasks. While these methods perform well in single-task learning, extending them to multi-task learning exacerbates common challenges, such as task interference and negative transfer, due to the limited number of trainable parameters. To address these issues, we introduce progressive task-specific multi-task adaptation, a novel parameter-efficient approach for multi-task learning. This approach introduces adapter modules in a pre-trained model such that these modules are shared across all tasks in the initial layers and become progressively more task-specific in the later layers. The motivation is to reduce the conflicts among tasks by allowing transfer learning across all tasks in the initial layers and enabling task-specific learning toward the prediction heads. Additionally, we propose a gradient-based approach for computing task similarity and use this measure to allocate similar tasks to the shared adapter modules. Our task similarity method introduces minimal overhead in the pipeline. We evaluate our approach by adapting the Swin Transformer for dense prediction tasks. Experiments on the PASCAL and NYUD-v2 datasets demonstrate that our approach outperforms a fully fine-tuned multi-task model while requiring only one-fifth of the trainable parameters. This approach achieves better relative improvement to single-task fine-tuning while reducing the number of trainable parameters and surpasses the current state-of-the-art methods for parameter-efficient multi-task learning.
Abstract (translated)
参数高效的微调方法已成为将预训练模型适应各种下游任务的有希望的解决方案。尽管这些方法在单一任务学习中表现出色，但将其扩展到多任务学习时，由于可训练参数数量有限，会加剧诸如任务干扰和负向迁移等常见挑战。为了解决这些问题，我们引入了一种新颖的参数高效多任务适应方法——渐进式特定任务多任务自适应。该方法在预训练模型中引入适配器模块，并且这些模块在初始层被所有任务共享，在后续层逐渐变得更具任务特异性。其动机在于通过允许跨所有任务的迁移学习来减少任务间的冲突，同时使预测头朝向任务特定的学习。此外，我们提出了一种基于梯度的方法来计算任务相似性，并使用这种方法将相似的任务分配给共享适配器模块。我们的任务相似性方法在管道中引入了最小开销。 我们在密集预测任务上通过微调Swin Transformer来评估此方法。实验结果表明，在PASCAL和NYUD-v2数据集上的性能优于完全微调的多任务模型，而所需训练参数仅为后者的五分之一。该方法相对于单一任务微调取得了更好的相对改进，并且在减少可训练参数数量的同时超越了当前最先进的参数高效多任务学习方法。
URL
https://arxiv.org/abs/2509.19602
