Abstract
Fine-tuning large-scale pre-trained models with limited data presents significant challenges for generalization. While Sharpness-Aware Minimization (SAM) has proven effective in improving generalization by seeking flat minima, its substantial extra memory and computation overhead make it impractical for large models. Integrating SAM with parameter-efficient fine-tuning methods like Low-Rank Adaptation (LoRA) is a promising direction. However, we find that directly applying SAM to LoRA parameters limits the sharpness optimization to a restricted subspace, hindering its effectiveness. To address this limitation, we propose Bi-directional Low-Rank Adaptation (Bi-LoRA), which introduces an auxiliary LoRA module to model SAM's adversarial weight perturbations. It decouples SAM's weight perturbations from LoRA optimization: the primary LoRA module adapts to specific tasks via standard gradient descent, while the auxiliary module captures the sharpness of the loss landscape through gradient ascent. Such dual-module design enables Bi-LoRA to capture broader sharpness for achieving flatter minima while remaining memory-efficient. Another important benefit is that the dual design allows for simultaneous optimization and perturbation, eliminating SAM's doubled training costs. Extensive experiments across diverse tasks and architectures demonstrate Bi-LoRA's efficiency and effectiveness in enhancing generalization.
Abstract (translated)
在数据有限的情况下，对大规模预训练模型进行微调以实现泛化是一项重大挑战。虽然敏锐度感知最小化（Sharpness-Aware Minimization, SAM）通过寻找平坦的极小值来提高泛化的有效性已经得到了证明，但其显著增加的记忆和计算开销使其在大型模型上的应用变得不切实际。将SAM与低秩适应（Low-Rank Adaptation, LoRA）等参数高效微调方法结合是一种有前景的方向。然而，我们发现直接将SAM应用于LoRA参数会限制尖锐度优化到一个受限的子空间中，从而影响其效果。 为了解决这一局限性，我们提出了一种双向低秩适应（Bi-directional Low-Rank Adaptation, Bi-LoRA）方法，通过引入辅助的LoRA模块来模拟SAM的对抗权重扰动。这种方法将SAM的权重扰动与LoRA优化过程解耦：主LoRA模块通过标准梯度下降法针对特定任务进行调整，而辅助模块则通过梯度上升捕捉损失地形图的锐利性。这种双模块设计使Bi-LoRA能够在保持内存效率的同时捕获更广泛的锐度以实现更平坦的极小值。此外，双重设计还允许同时进行优化和扰动，消除了SAM加倍训练成本的问题。 跨多种任务和架构的大量实验表明了Bi-LoRA在提高泛化能力方面既高效又有效。
URL
https://arxiv.org/abs/2508.19564
