Abstract
Edge computing allows artificial intelligence and machine learning models to be deployed on edge devices, where they can learn from local data and collaborate to form a global model. Federated learning (FL) is a distributed machine learning technique that facilitates this process while preserving data privacy. However, FL also faces challenges such as high computational and communication costs regarding resource-constrained devices, and poor generalization performance due to the heterogeneity of data across edge clients and the presence of out-of-distribution data. In this paper, we propose the Gradient-Congruity Guided Federated Sparse Training (FedSGC), a novel method that integrates dynamic sparse training and gradient congruity inspection into federated learning framework to address these issues. Our method leverages the idea that the neurons, in which the associated gradients with conflicting directions with respect to the global model contain irrelevant or less generalized information for other clients, and could be pruned during the sparse training process. Conversely, the neurons where the associated gradients with consistent directions could be grown in a higher priority. In this way, FedSGC can greatly reduce the local computation and communication overheads while, at the same time, enhancing the generalization abilities of FL. We evaluate our method on challenging non-i.i.d settings and show that it achieves competitive accuracy with state-of-the-art FL methods across various scenarios while minimizing computation and communication costs.
Abstract (translated)
边缘计算允许人工智能和机器学习模型在边缘设备上部署，从本地数据中学习并协同形成全局模型。联邦学习（FL）是一种分布式机器学习技术，它通过保留数据隐私来促进这一过程。然而，FL也面临着一些挑战，如资源受限设备的计算和通信成本较高，以及由于边缘客户端数据异质性和存在离散数据而导致的泛化性能较差。在本文中，我们提出了 Gradient-Congruity Guided Federated Sparse Training (FedSGC) 方法，一种将动态稀疏训练和梯度一致性检查集成到联邦学习框架中的新方法，以解决这些问题。我们的方法利用了神经元中与全局模型相关但方向不一致的梯度包含无关或较少泛化信息的假设，并可以在稀疏训练过程中进行剪枝。相反，与全局模型方向一致的梯度可以以更高的优先级进行生长。这样，FedSGC 可以在降低本地计算和通信开销的同时，增强 FL 的泛化能力。我们在具有挑战性的非均匀设置中评估了我们的方法，结果表明，它在不同场景下的竞争精度与最先进的 FL 方法相当，同时最小化计算和通信成本。
URL
https://arxiv.org/abs/2405.01189
