Abstract
Estimating the network performance using zero-cost (ZC) metrics has proven both its efficiency and efficacy in Neural Architecture Search (NAS). However, a notable limitation of most ZC proxies is their inconsistency, as reflected by the substantial variation in their performance across different problems. Furthermore, the design of existing ZC metrics is manual, involving a time-consuming trial-and-error process that requires substantial domain expertise. These challenges raise two critical questions: (1) Can we automate the design of ZC metrics? and (2) Can we utilize the existing hand-crafted ZC metrics to synthesize a more generalizable one? In this study, we propose a framework based on Symbolic Regression via Genetic Programming to automate the design of ZC metrics. Our framework is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to true network performance across diverse NAS search spaces and tasks. Extensive experiments on 13 problems from NAS-Bench-Suite-Zero demonstrate that our automatically generated proxies consistently outperform hand-crafted alternatives. Using our evolved proxy metric as the search objective in an evolutionary algorithm, we could identify network architectures with competitive performance within 15 minutes using a single consumer GPU.
Abstract (translated)
利用零成本（Zero-Cost，ZC）度量估算网络性能在神经架构搜索（NAS）中已被证明具有高效性和有效性。然而，大多数ZC代理的一个显著限制是其不一致性，这体现在它们在不同问题上的表现波动较大上。此外，现有ZC度量的设计过程是手动完成的，并且涉及耗时的试错过程，需要大量领域专业知识。这些问题提出了两个关键问题：（1）我们能否自动化设计ZC度量？（2）我们是否可以利用现有的手工制作的ZC度量来合成一个更具有普适性的度量？ 在本研究中，我们提出了一种基于遗传编程进行符号回归的框架，用于自动设计ZC度量。我们的框架不仅高度可扩展，而且能够快速生成一种零成本度量，在各种NAS搜索空间和任务上与真实网络性能之间具有强烈的正向等级相关性。 通过在来自NAS-Bench-Suite-Zero套件中的13个问题上的广泛实验表明，我们自动产生的代理始终优于手工设计的替代方案。使用我们的演化出来的代理指标作为进化算法中的搜索目标，在单个消费级GPU上仅需15分钟即可识别出具有竞争力性能的网络架构。
URL
https://arxiv.org/abs/2505.15832
