Abstract
Neural architecture search (NAS) faces a challenge in balancing the exploration of expressive, broad search spaces that enable architectural innovation with the need for efficient evaluation of architectures to effectively search such spaces. We investigate surrogate model training for improving search in highly expressive NAS search spaces based on context-free grammars. We show that i) surrogate models trained either using zero-cost-proxy metrics and neural graph features (GRAF) or by fine-tuning an off-the-shelf LM have high predictive power for the performance of architectures both within and across datasets, ii) these surrogates can be used to filter out bad architectures when searching on novel datasets, thereby significantly speeding up search and achieving better final performances, and iii) the surrogates can be further used directly as the search objective for huge speed-ups.
Abstract (translated)
神经架构搜索（NAS）在平衡探索表达丰富、广泛的搜索空间以实现架构创新和高效评估架构的需求之间面临着挑战。我们研究了基于上下文无关语法的高度表现力的NAS搜索空间中，通过训练替代模型来改进搜索的方法。我们的发现包括： 1. 使用零成本代理指标或神经图特征（GRAF）训练的替代模型，以及微调现成的语言模型得到的替代模型，对于预测架构在数据集内部和跨不同数据集上的性能具有很高的预测能力。 2. 这些替代模型可以用于过滤新数据集中较差的架构，从而显著加快搜索速度并达到更好的最终性能。 3. 此外，这些替代模型可以直接用作搜索目标，以实现巨大的加速效果。
URL
https://arxiv.org/abs/2504.12971
