Abstract
Although Federated Learning has been widely studied in recent years, there are still high overhead expenses in each communication round for large-scale models such as Vision Transformer. To lower the communication complexity, we propose a novel Federated Block Coordinate Gradient Descent (FedBCGD) method for communication efficiency. The proposed method splits model parameters into several blocks, including a shared block and enables uploading a specific parameter block by each client, which can significantly reduce communication overhead. Moreover, we also develop an accelerated FedBCGD algorithm (called FedBCGD+) with client drift control and stochastic variance reduction. To the best of our knowledge, this paper is the first work on parameter block communication for training large-scale deep models. We also provide the convergence analysis for the proposed algorithms. Our theoretical results show that the communication complexities of our algorithms are a factor $1/N$ lower than those of existing methods, where $N$ is the number of parameter blocks, and they enjoy much faster convergence than their counterparts. Empirical results indicate the superiority of the proposed algorithms compared to state-of-the-art algorithms. The code is available at this https URL.
Abstract (translated)
尽管联邦学习（Federated Learning）近年来得到了广泛的研究，但对于大规模模型如视觉变换器（Vision Transformer），每次通信轮次中仍然存在高昂的开销。为了降低通信复杂性，我们提出了一种新的、旨在提高通信效率的联邦块坐标梯度下降法（FedBCGD）。该方法将模型参数分割成若干块，包括一个共享块，并允许每个客户端上传特定参数块，这可以显著减少通信开销。此外，我们还开发了一种加速版FedBCGD算法（称为FedBCGD+），它结合了客户端漂移控制和随机方差减小技术。 据我们所知，本文是关于用于训练大规模深度模型的参数块通信的第一项工作。我们也提供了对所提出算法的收敛性分析。我们的理论结果表明，与现有方法相比，我们算法的通信复杂度低了一个因子$1/N$（其中N为参数块的数量），并且它们比现有的方法具有更快的收敛速度。实验结果显示了所提出的算法相较于最先进的算法的优势。代码可在提供的链接中获取：[此处应插入实际URL]。
URL
https://arxiv.org/abs/2603.05116
