Abstract
Adapting pre-trained vision models using parameter-efficient fine-tuning (PEFT) remains challenging, as it aims to achieve performance comparable to full fine-tuning using a minimal number of trainable parameters. When applied to complex dense prediction tasks, existing methods exhibit limitations, including input-agnostic modeling and redundant cross-layer representations. To this end, we propose AdaRoute, a new adapter-style method featuring a simple mixture-of-experts (MoE) architecture. Specifically, we introduce shared expert centers, where each expert is a trainable parameter matrix. During a feedforward pass, each AdaRoute module in the network dynamically generates weight matrices tailored for the current module via a simple dynamic parameter routing mechanism, which selectively aggregates parameter matrices in the corresponding expert center. Dynamic weight matrices in AdaRoute modules facilitate low-rank adaptation in an input-dependent manner, thus generating more customized and powerful feature representations. Moreover, since AdaRoute modules across multiple network layers share the same expert center, they improve feature diversity by promoting implicit cross-layer feature interaction. Extensive experiments demonstrate the superiority of AdaRoute on diverse vision tasks, including semantic segmentation, object detection and instance segmentation, and panoptic segmentation. Code will be available at: this https URL.
Abstract (translated)
使用参数高效微调（Parameter-Efficient Fine-Tuning，PEFT）来调整预训练视觉模型仍然具有挑战性，因为其目标是在使用最少可训练参数的情况下实现与全量微调相当的性能。当应用于复杂的密集预测任务时，现有的方法表现出一些局限性，包括对输入不敏感建模和跨层冗余表示的问题。为此，我们提出了一种新的适配器风格的方法——AdaRoute，它采用简单的混合专家（Mixture-of-Experts, MoE）架构。 具体来说，我们在 AdaRoute 中引入了共享的专家中心，每个专家是一个可训练参数矩阵。在网络前向传播过程中，每个 AdaRoute 模块通过一个简单的动态参数路由机制自适应地生成适合当前模块的权重矩阵，并从相应的专家中心选择性聚合参数矩阵。AdaRoute 模块中的动态权重矩阵以输入依赖的方式促进了低秩调整，从而产生了更加定制化且强大的特征表示。 此外，由于网络各层之间的 AdaRoute 模块共享相同的专家中心，它们通过促进隐式跨层特征交互提高了特征多样性。广泛的实验表明，AdaRoute 在包括语义分割、目标检测和实例分割以及全景分割在内的多种视觉任务上表现出优越性能。 代码将发布在以下链接：[提供给定的 URL]
URL
https://arxiv.org/abs/2602.06862
