Abstract
Neural architecture search (NAS) is a challenging problem. Hierarchical search spaces allow for cheap evaluations of neural network sub modules to serve as surrogate for architecture evaluations. Yet, sometimes the hierarchy is too restrictive or the surrogate fails to generalize. We present FaDE which uses differentiable architecture search to obtain relative performance predictions on finite regions of a hierarchical NAS space. The relative nature of these ranks calls for a memory-less, batch-wise outer search algorithm for which we use an evolutionary algorithm with pseudo-gradient descent. FaDE is especially suited on deep hierarchical, respectively multi-cell search spaces, which it can explore by linear instead of exponential cost and therefore eliminates the need for a proxy search space. Our experiments show that firstly, FaDE-ranks on finite regions of the search space correlate with corresponding architecture performances and secondly, the ranks can empower a pseudo-gradient evolutionary search on the complete neural architecture search space.
Abstract (translated)
神经架构搜索（NAS）是一个具有挑战性的问题。分层搜索空间允许对神经网络子模块进行廉价的评估，作为架构评估的代理。然而，有时候分层结构过于严格，或者代理无法泛化。我们提出了FaDE，它使用不同的iable架构搜索来获得分层 NAS 空间中有限区域的相对性能预测。这些相对排名的性质要求我们使用进化算法（我们使用具有伪梯度的进化算法）进行无记忆、批量的外搜索。FaDE 特别适用于具有深度分层和多细胞搜索空间的NAS，通过线性成本而不是指数成本进行探索，因此无需代理搜索空间。我们的实验结果表明，首先，FaDE在搜索空间有限区域上的排名与相应的架构性能相关联，其次，排名可以推动在完整神经架构搜索空间上的伪梯度进化搜索。
URL
https://arxiv.org/abs/2404.16218
