Abstract
Neural architecture search (NAS) has shown promise towards automating neural network design for a given task, but it is computationally demanding due to training costs associated with evaluating a large number of architectures to find the optimal one. To speed up NAS, recent works limit the search to network building blocks (modular search) instead of searching the entire architecture (global search), approximate candidates' performance evaluation in lieu of complete training, and use gradient descent rather than naturally suitable discrete optimization approaches. However, modular search does not determine network's macro architecture i.e. depth and width, demanding manual trial and error post-search, hence lacking automation. In this work, we revisit NAS and design a navigable, yet architecturally diverse, macro-micro search space. In addition, to determine relative rankings of candidates, existing methods employ consistent approximations across entire search spaces, whereas different networks may not be fairly comparable under one training protocol. Hence, we propose an architecture-aware approximation with variable training schemes for different networks. Moreover, we develop an efficient search strategy by disjoining macro-micro network design that yields competitive architectures in terms of both accuracy and size. Our proposed framework achieves a new state-of-the-art on EMNIST and KMNIST, while being highly competitive on the CIFAR-10, CIFAR-100, and FashionMNIST datasets and being 2-4x faster than the fastest global search methods. Lastly, we demonstrate the transferability of our framework to real-world computer vision problems by discovering competitive architectures for face recognition applications.
Abstract (translated)
神经架构搜索（NAS）在自动化特定任务的神经网络设计方面显示出潜力，但由于评估大量架构以找到最优架构时涉及高昂的训练成本，这一过程计算密集。为了加速NAS，最近的工作将搜索限制在网络构建模块上（模块化搜索），而不是在整个架构中进行搜索（全局搜索），并且通过近似候选者的性能评价来替代完整训练，并使用梯度下降而非更适合的离散优化方法。然而，模块化搜索无法确定网络的整体结构即深度和宽度，在搜索后仍需手动试验调整，从而失去了自动化特性。 在这项工作中，我们重新审视了NAS设计了一个可导航且架构多样的宏-微搜索空间。此外，现有方法为了确定候选者的相对排名会在整个搜索空间中使用一致的近似法，但不同的网络在相同的训练协议下可能无法公平比较。因此，我们提出了基于架构感知的不同网络采用不同训练方案的近似策略。此外，通过分离宏-微网络设计，我们开发了一种高效的搜索策略，在准确性和大小方面都产生了竞争性的架构。我们的框架在EMNIST和KMNIST数据集上达到了新的SOTA（State of the Art），同时在CIFAR-10、CIFAR-100以及FashionMNIST数据集中表现出高度竞争力，并且比最快的全局搜索方法快2到4倍。 最后，我们通过为面部识别应用发现竞争性架构展示了框架向现实世界计算机视觉问题的可转移能力。
URL
https://arxiv.org/abs/2502.03553
