Abstract
Training-free Neural Architecture Search (NAS) efficiently identifies high-performing neural networks using zero-cost (ZC) proxies. Unlike multi-shot and one-shot NAS approaches, ZC-NAS is both (i) time-efficient, eliminating the need for model training, and (ii) interpretable, with proxy designs often theoretically grounded. Despite rapid developments in the field, current SOTA ZC proxies are typically constrained to well-established convolutional search spaces. With the rise of Large Language Models shaping the future of deep learning, this work extends ZC proxy applicability to Vision Transformers (ViTs). We present a new benchmark using the Autoformer search space evaluated on 6 distinct tasks and propose Layer-Sample Wise Activation with Gradients information (L-SWAG), a novel, generalizable metric that characterizes both convolutional and transformer architectures across 14 tasks. Additionally, previous works highlighted how different proxies contain complementary information, motivating the need for a ML model to identify useful combinations. To further enhance ZC-NAS, we therefore introduce LIBRA-NAS (Low Information gain and Bias Re-Alignment), a method that strategically combines proxies to best represent a specific benchmark. Integrated into the NAS search, LIBRA-NAS outperforms evolution and gradient-based NAS techniques by identifying an architecture with a 17.0% test error on ImageNet1k in just 0.1 GPU days.
Abstract (translated)
无训练神经架构搜索（NAS）能够高效地识别高性能的神经网络，而无需使用任何模型训练。与多次射击和单次射击的NAS方法不同，零成本（ZC）NAS具有两个优点：一是时间效率高，不需要进行模型训练；二是可解释性强，代理设计通常有坚实的理论基础。尽管该领域的发展迅速，但当前最先进的ZC代理大多局限于成熟的卷积搜索空间内。随着大型语言模型塑造深度学习的未来，这项工作将ZC代理的应用范围扩展到了视觉变压器（ViTs）。我们提出了一项新的基准测试，采用Autoformer搜索空间，并在6个不同的任务上进行评估。此外，我们还提出了Layer-Sample Wise Activation with Gradients信息（L-SWAG），这是一种全新的、可推广的度量标准，能够表征卷积和转换器架构之间的关系，涵盖了14项任务。 先前的研究强调了不同代理之间互补信息的重要性，并指出需要机器学习模型来识别有用的组合。为了进一步提高ZC-NAS的性能，我们引入了LIBRA-NAS（低信息增益与偏差再对准）方法，该方法战略性地结合各种代理以最佳方式表示特定基准测试。在NAS搜索中集成LIBRA-NAS后，在仅使用0.1个GPU天的时间里，它就能识别出一个在ImageNet1k上具有17.0%测试误差的架构，并且其性能超过了基于进化和梯度的方法。
URL
https://arxiv.org/abs/2505.07300
