Abstract
Since random initial models in Federated Learning (FL) can easily result in unregulated Stochastic Gradient Descent (SGD) processes, existing FL methods greatly suffer from both slow convergence and poor accuracy, especially for non-IID scenarios. To address this problem, we propose a novel FL method named CyclicFL, which can quickly derive effective initial models to guide the SGD processes, thus improving the overall FL training performance. Based on the concept of Continual Learning (CL), we prove that CyclicFL approximates existing centralized pre-training methods in terms of classification and prediction performance. Meanwhile, we formally analyze the significance of data consistency between the pre-training and training stages of CyclicFL, showing the limited Lipschitzness of loss for the pre-trained models by CyclicFL. Unlike traditional centralized pre-training methods that require public proxy data, CyclicFL pre-trains initial models on selected clients cyclically without exposing their local data. Therefore, they can be easily integrated into any security-critical FL methods. Comprehensive experimental results show that CyclicFL can not only improve the classification accuracy by up to 16.21%, but also significantly accelerate the overall FL training processes.
Abstract (translated)
由于联邦学习(FL)中的随机初始模型很容易导致无控制的自由梯度下降(SGD)过程,现有的FL方法在缓慢收敛和低精度方面 greatly suffer,特别是针对非独立数据集的情况。为了解决这一问题,我们提出了一种名为CyclicFL的新型FL方法,它可以快速生成有效的初始模型,以指导SGD过程,从而改善整个FL训练性能。基于持续学习(CL)的概念,我们证明了CyclicFL在分类和预测性能方面的近似性,并 formal 分析了CyclicFL的预训练阶段和训练阶段的数据一致性,表明CyclicFL对预训练模型的损失限制有限。与传统的中心化预训练方法需要公共代理数据不同,CyclicFL通过循环选择特定的客户,定期预训练初始模型,而不会暴露客户的本地数据。因此,它们可以轻松地集成到任何具有安全性关键问题的FL方法中。全面的实验结果表明,CyclicFL不仅可以提高分类精度高达16.21%,而且还显著加速了整个FL训练过程。
URL
https://arxiv.org/abs/2301.12193
