Abstract
Federated Learning is a distributed machine learning environment, which ensures that clients complete collaborative training without sharing private data, only by exchanging parameters. However, the data does not satisfy the same distribution and the computing resources of clients are different, which brings challenges to the related research. To better solve the above heterogeneous problems, we designed a novel federated learning method. The local model consists of the pre-trained model as the backbone and fully connected layers as the head. The backbone can extract features for the head, and the embedding vector of classes is shared between clients to optimize the head so that the local model can perform better. By sharing the embedding vector of classes, instead of parameters based on gradient space, clients can better adapt to private data, and it is more efficient in the communication between the server and clients. To better protect privacy, we proposed a privacy-preserving hybrid method to add noise to the embedding vector of classes, which has less impact on the local model performance under the premise of satisfying differential privacy. We conduct a comprehensive evaluation with other federated learning methods on the self-built vehicle dataset under non-independent identically distributed(Non-IID)
Abstract (translated)
分布式学习是一种分布式机器学习环境,通过交换参数来实现 clients 之间的合作训练,而无需共享私人数据。然而,数据分布不符合相同的模式, clients 的计算资源也各不相同,这给相关研究带来了挑战。为了解决这些问题,我们设计了一种新分布式学习方法。该本地模型由预先训练模型作为骨架,完全连接层作为头部。骨架可以提取头部的特征,并将类别嵌入向量在不同 clients 之间共享,以优化头部,从而使本地模型表现更好。通过共享类别嵌入向量,而不是基于梯度空间的参数,Clients 可以更好地适应私人数据,并且服务器和客户之间的通信更加高效。为了更好地保护隐私,我们提出了一种隐私 preserving 混合方法,用于在类别嵌入向量中添加噪声,在满足差异隐私的前提下,该方法对本地模型表现的影响较小。我们与其他分布式学习方法一起评估了自建立车辆数据集,该数据集在独立同分布(IID)条件下不存在相同的个体(个体ID)。
URL
https://arxiv.org/abs/2301.11705
