Abstract
Modern neural architecture search (NAS) is inherently multi-objective, balancing trade-offs such as accuracy, parameter count, and computational cost. This complexity makes NAS computationally expensive and nearly impossible to solve without efficient approximations. To address this, we propose a novel surrogate modelling approach that leverages an ensemble of Siamese network blocks to predict dominance relationships between candidate architectures. Lightweight and easy to train, the surrogate achieves 92% accuracy and replaces the crowding distance calculation in the survivor selection strategy with a heuristic rule based on model size. Integrated into a framework termed SiamNAS, this design eliminates costly evaluations during the search process. Experiments on NAS-Bench-201 demonstrate the framework's ability to identify Pareto-optimal solutions with significantly reduced computational costs. The proposed SiamNAS identified a final non-dominated set containing the best architecture in NAS-Bench-201 for CIFAR-10 and the second-best for ImageNet, in terms of test error rate, within 0.01 GPU days. This proof-of-concept study highlights the potential of the proposed Siamese network surrogate model to generalise to multi-tasking optimisation, enabling simultaneous optimisation across tasks. Additionally, it offers opportunities to extend the approach for generating Sets of Pareto Sets (SOS), providing diverse Pareto-optimal solutions for heterogeneous task settings.
Abstract (translated)
现代神经架构搜索（NAS）本质上是多目标的，需要在准确性、参数数量和计算成本之间进行权衡。这种复杂性使得NAS计算成本高昂，并且几乎无法没有高效近似的情况下解决。为了应对这一挑战，我们提出了一种新颖的代理建模方法，该方法利用Siamese网络模块的集合来预测候选架构之间的支配关系。轻量级且易于训练的代理模型在准确性方面达到了92%，并且用基于模型大小的启发式规则替代了生存选择策略中的拥挤距离计算。集成到一个名为SiamNAS的框架中，这一设计消除了搜索过程中昂贵的评估。 在NAS-Bench-201上的实验展示了该框架能够在显著降低计算成本的情况下识别帕累托最优解的能力。所提出的SiamNAS在不到0.01个GPU天的时间里，在CIFAR-10上识别了NAS-Bench-201中最佳架构，并且在ImageNet上找到了第二佳的架构，根据测试错误率来衡量。 这项概念验证研究展示了提议的Siamese网络代理模型泛化到多任务优化中的潜力，使跨任务的同时优化成为可能。此外，它还为生成集合帕累托集（SOS）提供了机会，在异构的任务设定中提供多样化的帕累托最优解。
URL
https://arxiv.org/abs/2506.02623
