Abstract
Federated learning ensures the privacy of clients by conducting distributed training on individual client devices and sharing only the model weights with a central server. However, in real-world scenarios, the heterogeneity of data among clients necessitates appropriate personalization methods. In this paper, we aim to address this heterogeneity using a form of parameter decoupling known as representation learning. Representation learning divides deep learning models into 'base' and 'head' components. The base component, capturing common features across all clients, is shared with the server, while the head component, capturing unique features specific to individual clients, remains local. We propose a new representation learning-based approach that suggests decoupling the entire deep learning model into more densely divided parts with the application of suitable scheduling methods, which can benefit not only data heterogeneity but also class heterogeneity. In this paper, we compare and analyze two layer scheduling approaches, namely forward (\textit{Vanilla}) and backward (\textit{Anti}), in the context of data and class heterogeneity among clients. Our experimental results show that the proposed algorithm, when compared to existing personalized federated learning algorithms, achieves increased accuracy, especially under challenging conditions, while reducing computation costs.
Abstract (translated)
联邦学习通过在个人设备上进行分布式训练并仅将模型权重与中央服务器共享来确保客户的隐私。然而,在现实场景中,数据在客户端之间的异质性需要适当的数据个性化方法。在本文中,我们使用一种称为表示学习的形式来解决异质性。表示学习将深度学习模型划分为“基础”和“头”组件。基础组件捕捉所有客户端共有的特征,与服务器共享;头组件捕捉单个客户端独特的特征,保留在本地。我们提出了一种新的表示学习-基于的方法,通过应用适当的调度方法,将整个深度学习模型划分为更密集的部分。这种方法不仅可以解决数据异质性,还可以减少类异质性。本文在客户端之间数据和类异质性的背景下,比较和分析了两种层调度方法(即前向(“Vanilla”)和后向(“Anti”))。我们的实验结果表明,与现有的个性化联邦学习算法相比,所提出的算法在数据和类异质性方面具有更高的准确度,尤其是在具有挑战性条件的场景下,同时减少了计算成本。
URL
https://arxiv.org/abs/2404.17799
