Abstract
It is commonly believed that gradient compression in federated learning (FL) enjoys significant improvement in communication efficiency with negligible performance degradation. In this paper, we find that gradient compression induces sharper loss landscapes in federated learning, particularly under non-IID data distributions, which suggests hindered generalization capability. The recently emerging Sharpness Aware Minimization (SAM) effectively searches for a flat minima by incorporating a gradient ascent step (i.e., perturbing the model with gradients) before the celebrated stochastic gradient descent. Nonetheless, the direct application of SAM in FL suffers from inaccurate estimation of the global perturbation due to data heterogeneity. Existing approaches propose to utilize the model update from the previous communication round as a rough estimate. However, its effectiveness is hindered when model update compression is incorporated. In this paper, we propose FedSynSAM, which leverages the global model trajectory to construct synthetic data and facilitates an accurate estimation of the global perturbation. The convergence of the proposed algorithm is established, and extensive experiments are conducted to validate its effectiveness.
Abstract (translated)
人们普遍认为，在联邦学习（FL）中，梯度压缩可以显著提高通信效率，并且几乎不会导致性能下降。然而，在本文中，我们发现梯度压缩在联邦学习中会导致更尖锐的损失景观，尤其是在非独立同分布（non-IID）的数据分布下，这表明泛化能力受到了阻碍。 最近出现的敏锐度感知最小化（SAM）通过在传统的随机梯度下降之前增加一个梯度上升步骤（即用梯度扰动模型），有效地寻找平坦的极小值。然而，在联邦学习中直接应用SAM会因为数据异质性而导致全局扰动估计不准确的问题。现有方法建议使用上一轮通信中的模型更新作为粗略估计，但当引入模型更新压缩时，这种方法的有效性就会受到限制。 在本文中，我们提出了一种名为FedSynSAM的方法，该方法利用全球模型轨迹构建合成数据，并促进对全局扰动的精确估计。我们建立了所提算法的收敛性理论，并通过广泛的实验验证了其有效性。
URL
https://arxiv.org/abs/2602.11584
