Abstract
Federated Learning has become a widely-used framework which allows learning a global model on decentralized local datasets under the condition of protecting local data privacy. However, federated learning faces severe optimization difficulty when training samples are not independently and identically distributed (non-i.i.d.). In this paper, we point out that the client sampling practice plays a decisive role in the aforementioned optimization difficulty. We find that the negative client sampling will cause the merged data distribution of currently sampled clients heavily inconsistent with that of all available clients, and further make the aggregated gradient unreliable. To address this issue, we propose a novel learning rate adaptation mechanism to adaptively adjust the server learning rate for the aggregated gradient in each round, according to the consistency between the merged data distribution of currently sampled clients and that of all available clients. Specifically, we make theoretical deductions to find a meaningful and robust indicator that is positively related to the optimal server learning rate and can effectively reflect the merged data distribution of sampled clients, and we utilize it for the server learning rate adaptation. Extensive experiments on multiple image and text classification tasks validate the great effectiveness of our method.
Abstract (translated)
分散学习已经成为一种广泛应用的框架,可以在保护本地数据隐私的条件下,学习分布式本地数据集的全球模型。然而,当训练样本不独立且同样分布(非i.i.d.)时,分散学习面临严重的优化难题。在本文中,我们指出,客户采样实践在上述优化难题中起到了决定作用。我们发现,消极客户采样将会导致当前采样客户的合并数据分布与所有可用客户的数据分布严重不一致,并进一步导致聚合梯度的不可靠性。为了解决这一问题,我们提出了一种新的学习速率适应机制,根据当前采样客户和所有可用客户的合并数据分布之间的一致性,自适应地调整服务器学习速率,在每个Round中。具体来说,我们进行理论推导,以找到一种有意义且稳健的指示器,它与最优服务器学习速率呈正相关,能够有效反映采样客户合并数据的分布,因此我们将其用于服务器学习速率适应。对多个图像和文本分类任务进行的广泛实验验证了我们方法的高效性。
URL
https://arxiv.org/abs/2301.10400
