Abstract
Zero-shot Neural Architecture Search (NAS) typically optimises the architecture search process by exploiting the network or gradient properties at initialisation through zero-cost proxies. The existing proxies often rely on labelled data, which is usually unavailable in real-world settings. Furthermore, the majority of the current methods focus either on optimising the convergence and generalisation attributes or solely on the expressivity of the network architectures. To address both limitations, we first demonstrate how channel collinearity affects the convergence and generalisation properties of a neural network. Then, by incorporating the convergence, generalisation and expressivity in one approach, we propose a zero-cost proxy that omits the requirement of labelled data for its computation. In particular, we leverage the Singular Value Decomposition (SVD) of the neural network layer features and the extrinsic curvature of the network output to design our proxy. %As a result, the proposed proxy is formulated as the simplified harmonic mean of the logarithms of two key components: the sum of the inverse of the feature condition number and the extrinsic curvature of the network output. Our approach enables accurate prediction of network performance on test data using only a single label-free data sample. Our extensive evaluation includes a total of six experiments, including the Convolutional Neural Network (CNN) search space, i.e. DARTS and the Transformer search space, i.e. AutoFormer. The proposed proxy demonstrates a superior performance on multiple correlation benchmarks, including NAS-Bench-101, NAS-Bench-201, and TransNAS-Bench-101-micro; as well as on the NAS task within the DARTS and the AutoFormer search space, all while being notably efficient. The code is available at this https URL.
Abstract (translated)
零样本神经架构搜索（NAS）通常通过在初始阶段利用网络或梯度属性的无成本代理来优化架构搜索过程。现有的许多代理依赖于有标签的数据，而在现实场景中这些数据往往不可用。此外，目前大多数方法要么侧重于优化收敛性和泛化属性，要么仅关注网络架构的表达能力。为了解决这两个限制，我们首先展示了通道共线性如何影响神经网络的收敛性和泛化特性。然后，通过将收敛性、泛化和表达能力结合在一个方法中，我们提出了一种无成本代理，该代理在计算时不依赖有标签的数据需求。具体来说，我们利用了神经网络层特征的奇异值分解（SVD）以及网络输出的外在曲率来设计我们的代理。 所提出的代理被公式化为两个关键组件对数的简化调和平均值：特征条件数之逆的总和与网络输出的外在曲率。这种方法仅使用一个无标签的数据样本就能准确预测网络在测试数据上的性能。我们进行了广泛的评估，包括六个实验，涉及卷积神经网络（CNN）搜索空间（例如DARTS）和Transformer搜索空间（如AutoFormer）。所提出的代理在多个相关基准上展示了优越的性能，包括NAS-Bench-101、NAS-Bench-201以及TransNAS-Bench-101-micro；同时，在DARTS和AutoFormer搜索空间内的NAS任务中表现同样出色，并且效率显著。代码可在该链接获得：[此处应填写URL]。
URL
https://arxiv.org/abs/2508.12977
