Abstract
Low-Rank Adaptation (LoRA), which introduces a product of two trainable low-rank matrices into frozen pre-trained weights, is widely used for efficient fine-tuning of language models in federated learning (FL). However, when combined with differentially private stochastic gradient descent (DP-SGD), LoRA faces substantial noise amplification: DP-SGD perturbs per-sample gradients, and the matrix multiplication of the LoRA update ($BA$) intensifies this effect. Freezing one matrix (e.g., $A$) reduces the noise but restricts model expressiveness, often resulting in suboptimal adaptation. To address this, we propose FedSVD, a simple yet effective method that introduces a global reparameterization based on singular value decomposition (SVD). In our approach, each client optimizes only the $B$ matrix and transmits it to the server. The server aggregates the $B$ matrices, computes the product $BA$ using the previous $A$, and refactorizes the result via SVD. This yields a new adaptive $A$ composed of the orthonormal right singular vectors of $BA$, and an updated $B$ containing the remaining SVD components. This reparameterization avoids quadratic noise amplification, while allowing $A$ to better capture the principal directions of the aggregate updates. Moreover, the orthonormal structure of $A$ bounds the gradient norms of $B$ and preserves more signal under DP-SGD, as confirmed by our theoretical analysis. As a result, FedSVD consistently improves stability and performance across a variety of privacy settings and benchmarks, outperforming relevant baselines under both private and non-private regimes.
Abstract (translated)
低秩适应（LoRA）通过将两个可训练的低秩矩阵的乘积引入冻结的预训练权重中，被广泛用于联邦学习（FL）中的语言模型高效微调。然而，在与差异隐私随机梯度下降（DP-SGD）结合时，LoRA面临着显著的噪声放大问题：DP-SGD会扰动每个样本的梯度，而LoRA更新的矩阵乘法($BA$)则增强了这一影响。冻结一个矩阵（例如 $A$）虽然可以减少噪音但限制了模型的表现力，往往导致次优适应。 为解决上述问题，我们提出了一种简单且有效的方法FedSVD，该方法基于奇异值分解（SVD）引入了一个全局重参数化。在我们的方法中，每个客户端仅优化矩阵 $B$ 并将其传输到服务器。服务器聚合了所有的 $B$ 矩阵，并使用之前的 $A$ 计算乘积 $BA$ ，然后通过 SVD 重新因子化该结果。这生成了一个新的适应性 $A$，由 $BA$ 的正交右奇异向量组成，并且更新后的 $B$ 包含剩余的 SVD 组件。这种重参数化避免了二次噪声放大，同时允许 $A$ 更好地捕捉聚合更新的主要方向。此外，由于 $A$ 的正交结构，它限制了 $B$ 的梯度范数并在 DP-SGD 下保留更多信号，这一点已被我们的理论分析所证实。 因此，在各种隐私设置和基准测试中，FedSVD 一致提高了稳定性和性能，并且在私人和非私人模式下均超越了相关的基线方法。
URL
https://arxiv.org/abs/2505.12805
