Abstract
Low-Rank Adaptation (LoRA) and its variants have delivered strong capability in Parameter-Efficient Transfer Learning (PETL) by minimizing trainable parameters and benefiting from reparameterization. However, their projection matrices remain unrestricted during training, causing high representation redundancy and diminishing the effectiveness of feature adaptation in the resulting subspaces. While existing methods mitigate this by manually adjusting the rank or implicitly applying channel-wise masks, they lack flexibility and generalize poorly across various datasets and architectures. Hence, we propose ReSoRA, a method that explicitly models redundancy between mapping subspaces and adaptively Regularizes Subspace redundancy of Low-Rank Adaptation. Specifically, it theoretically decomposes the low-rank submatrices into multiple equivalent subspaces and systematically applies de-redundancy constraints to the feature distributions across different projections. Extensive experiments validate that our proposed method consistently facilitates existing state-of-the-art PETL methods across various backbones and datasets in vision-language retrieval and standard visual classification benchmarks. Besides, as a training supervision, ReSoRA can be seamlessly integrated into existing approaches in a plug-and-play manner, with no additional inference costs. Code is publicly available at: this https URL.
Abstract (translated)
低秩适应（LoRA）及其变体在参数高效转移学习（PETL）中表现出强大的能力，通过减少可训练参数并利用重新参数化获益。然而，在训练过程中它们的投影矩阵仍然不受限制，导致表示冗余度高，并降低了所生成子空间中的特征适应效果。尽管现有的方法通过手动调整秩或隐式应用通道级掩码来缓解这一问题，但这些方法缺乏灵活性且在不同数据集和架构上泛化能力较差。 因此，我们提出了ReSoRA（Regularizing Subspace Redundancy of Low-Rank Adaptation），一种明确建模映射子空间之间冗余度并自适应地正则化低秩适应子空间冗余的方法。具体而言，它从理论上将低秩子矩阵分解为多个等价的子空间，并系统性地在不同投影下的特征分布上施加去冗余约束。 广泛的实验验证了我们提出的方法能够持续增强现有的最先进的PETL方法，在视觉-语言检索和标准视觉分类基准测试中的各种骨干网络和数据集上的表现。此外，作为训练监督，ReSoRA可以无缝集成到现有方法中，并且在不增加推理成本的情况下采用插件即用的方式。 代码可在以下网址获取：[this https URL]（请将URL替换为实际链接）。
URL
https://arxiv.org/abs/2507.20745
