Abstract
Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies, capturing network characteristics related to the final performance. However, network rankings estimated by previous training-free NAS methods have shown weak correlations with the performance. To address this issue, we propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance. To achieve this, we introduce four novel zero-cost proxies that are complementary to each other, analyzing distinct traits of architectures in the views of expressivity, progressivity, trainability, and complexity. The proxy scores can be obtained simultaneously within a single forward and backward pass, making an overall NAS process highly efficient. In order to integrate the rankings predicted by our proxies effectively, we introduce a non-linear ranking aggregation method that highlights the networks highly-ranked consistently across all the proxies. Experimental results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all while maintaining a reasonable runtime cost.
Abstract (translated)
无需训练的网络架构搜索（NAS）旨在通过零成本代理发现具有高性能的网络，并捕获与最终性能相关的网络特征。然而，之前基于NAS的方法估计的网络排名与性能之间存在较弱的关联。为了解决这个问题，我们提出了AZ-NAS，一种利用各种零成本代理的集合并提高预测网络排名与地面真值之间关联的新方法。为了解决这个问题，我们引入了四个新的零成本代理，这些代理在表现力、进步性、可训练性和复杂性方面具有互补性，分析架构的不同特征。代理分数可以在一次前向和反向传播中同时获得，使整体NAS过程非常高效。为了有效地整合我们的代理的排名，我们引入了一种非线性排名聚合方法，该方法突出了网络在所有代理中高度排名的一致性。实验结果充分证明了AZ-NAS的有效性和高效性，在标准基准测试中超过了最先进的方法，同时保持了合理的运行成本。
URL
https://arxiv.org/abs/2403.19232
