Abstract
Predictor-based methods have substantially enhanced Neural Architecture Search (NAS) optimization. The efficacy of these predictors is largely influenced by the method of encoding neural network architectures. While traditional encodings used an adjacency matrix describing the graph structure of a neural network, novel encodings embrace a variety of approaches from unsupervised pretraining of latent representations to vectors of zero-cost proxies. In this paper, we categorize and investigate neural encodings from three main types: structural, learned, and score-based. Furthermore, we extend these encodings and introduce \textit{unified encodings}, that extend NAS predictors to multiple search spaces. Our analysis draws from experiments conducted on over 1.5 million neural network architectures on NAS spaces such as NASBench-101 (NB101), NB201, NB301, Network Design Spaces (NDS), and TransNASBench-101. Building on our study, we present our predictor \textbf{FLAN}: \textbf{Fl}ow \textbf{A}ttention for \textbf{N}AS. FLAN integrates critical insights on predictor design, transfer learning, and \textit{unified encodings} to enable more than an order of magnitude cost reduction for training NAS accuracy predictors. Our implementation and encodings for all neural networks are open-sourced at \href{this https URL}{this https URL\_nas}.
Abstract (translated)
基于预测器的神经网络架构搜索（NAS）优化方法已经极大地增强了NAS。这些预测器的有效性很大程度上取决于编码神经网络架构的方法。虽然传统的编码方法使用邻接矩阵描述神经网络的图形结构，而新的编码方法则采用各种无监督预训练、零成本代理的方案，从神经网络的图结构编码到向量表示。在本文中，我们将分类并研究三种主要的神经编码：结构、学习到的和基于分数的。此外，我们将这些编码扩展到多个搜索空间，引入了统一编码，将NAS预测器扩展到多个搜索空间。我们的分析基于在NAS空间上超过1500万神经网络架构的实验，如NASBench-101（NB101）、NB201、NB301、网络设计空间（NDS）和TransNASBench-101。基于我们的研究，我们提出了预测器FLAN：FLow Attention for NAS。FLAN集成了关于预测器设计、迁移学习和统一编码的关键见解，以实现训练NAS准确度预测器超过一倍的成本降低。我们的实现和对所有神经网络的编码都是开源的，您可以点击以下链接访问：<https://this https URL>this https URL_nas。
URL
https://arxiv.org/abs/2403.02484
