Abstract
Federated Learning (FL) often struggles with data heterogeneity due to the naturally uneven distribution of user data across devices. Federated Neural Architecture Search (NAS) enables collaborative search for optimal model architectures tailored to heterogeneous data to achieve higher accuracy. However, this process is time-consuming due to extensive search space and retraining. To overcome this, we introduce FedMetaNAS, a framework that integrates meta-learning with NAS within the FL context to expedite the architecture search by pruning the search space and eliminating the retraining stage. Our approach first utilizes the Gumbel-Softmax reparameterization to facilitate relaxation of the mixed operations in the search space. We then refine the local search process by incorporating Model-Agnostic Meta-Learning, where a task-specific learner adapts both weights and architecture parameters (alphas) for individual tasks, while a meta learner adjusts the overall model weights and alphas based on the gradient information from task learners. Following the meta-update, we propose soft pruning using the same trick on search space to gradually sparsify the architecture, ensuring that the performance of the chosen architecture remains robust after pruning which allows for immediate use of the model without retraining. Experimental evaluations demonstrate that FedMetaNAS significantly accelerates the search process by more than 50\% with higher accuracy compared to FedNAS.
Abstract (translated)
联邦学习（FL）通常会因为用户数据在不同设备上的分布不均而面临数据异质性的问题。联邦神经架构搜索（NAS）通过协作搜索，旨在为异构数据找到最优的模型结构以提高准确性。然而，这一过程由于搜索空间广泛以及需要重新训练而耗时较长。为了克服这个问题，我们引入了FedMetaNAS框架，该框架将元学习与NAS集成在联邦学习环境中，通过缩小搜索空间和消除重训阶段来加速架构搜索。 我们的方法首先使用Gumbel-Softmax再参数化技术使搜索空间中的混合操作松弛。随后，我们利用模型无关的元学习来优化局部搜索过程，在这种过程中，特定任务的学习者同时调整权重和架构参数（alphas）以适应个体任务，而元学习者则根据来自各个任务学习者的梯度信息调整整个模型的权重和alphas。 在完成元更新后，我们提出了一种使用同样技巧进行软修剪的方法来逐渐稀疏化架构。这种方法确保了在剪枝后的选定架构性能仍然稳健，并且可以在不重新训练的情况下直接使用该模型。 实验评估表明，FedMetaNAS与FedNAS相比，在提高搜索过程的速度超过50%的同时还能获得更高的准确性。
URL
https://arxiv.org/abs/2504.06457
