Abstract
Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks. Mixture-of-LoRAs models expand neural networks efficiently by routing each layer input to a small subset of specialized LoRAs of the layer. Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router. Despite their empirical promise, we observe that the routing weights are typically extremely imbalanced across LoRAs in practice, where only one or two LoRAs often dominate the routing weights. This essentially limits the number of effective LoRAs and thus severely hinders the expressive power of existing Mixture-of-LoRAs models. In this work, we attribute this weakness to the nature of learnable routing weights and rethink the fundamental design of the router. To address this critical issue, we propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix). Our key idea is using non-learnable routing weights to ensure all active LoRAs to be equally effective, with no LoRA dominating the routing weights. However, our routers cannot be trained directly via gradient descent due to our non-learnable routing weights. Hence, we further propose an unbiased gradient estimator for the router by employing the reinforce leave-one-out (RLOO) technique, where we regard the supervision loss as the reward and the router as the policy in reinforcement learning. Our gradient estimator also enables to scale up training compute to boost the predictive performance of our ReMix. Extensive experiments demonstrate that our proposed ReMix significantly outperform state-of-the-art parameter-efficient finetuning methods under a comparable number of activated parameters.
Abstract (translated)
低秩适配器（LoRAs）是一种参数高效的微调技术，它通过将可训练的低秩矩阵注入预训练模型来使这些模型适应新任务。Mixture-of-LoRAs模型通过将每一层的输入路由到该层的一小部分专门化的LoRA中，高效地扩展神经网络。现有的Mixture-of-LoRAs路由器为每个LoRA分配一个可学习的路由权重，以实现端到端训练。 尽管这种方法在实践中表现出了潜力，但我们观察到，在实际应用中，这些路由权重通常会在不同LoRA之间极为不平衡，其中只有少数几个LoRA会主导整个路由权重。这实质上限制了有效LoRA的数量，从而严重阻碍了现有Mixture-of-LoRAs模型的表达能力。 为了解决这一关键问题，我们归因于可学习路由权重的本质，并重新审视路由器的基本设计。为此，我们提出了一种新的路由器设计方案，称为针对混合低秩适配器（Reinforcement Routing for Mixture-of-LoRAs, ReMix）。我们的主要思想是使用不可学习的路由权重来确保所有激活状态下的LoRA同等有效，没有单一的LoRA能够主导整个路由权重。 然而，由于我们使用的非可学习路由权重，直接通过梯度下降训练这种路由器是不可能的。因此，我们进一步提出了一种无偏导数估计器，通过采用强化学习中的Reinforce Leave-One-Out (RLOO)技术来实现，将监督损失视为奖励，而路由器则作为策略。我们的导数估计器还使计算资源能够扩展，从而提升我们ReMix的预测性能。 广泛的实验结果表明，在激活参数数量大致相等的情况下，所提出的ReMix方法显著优于现有的基于参数效率的微调方法。
URL
https://arxiv.org/abs/2603.10160
