Abstract
Neural architecture search (NAS) is a hard computationally expensive optimization problem with a discrete, vast, and spiky search space. One of the key research efforts dedicated to this space focuses on accelerating NAS via certain proxy evaluations of neural architectures. Different from the prevalent predictor-based methods using surrogate models and differentiable architecture search via supernetworks, we propose an optimization proxy to streamline the NAS as an end-to-end optimization framework, named OptiProxy-NAS. In particular, using a proxy representation, the NAS space is reformulated to be continuous, differentiable, and smooth. Thereby, any differentiable optimization method can be applied to the gradient-based search of the relaxed architecture parameters. Our comprehensive experiments on $12$ NAS tasks of $4$ search spaces across three different domains including computer vision, natural language processing, and resource-constrained NAS fully demonstrate the superior search results and efficiency. Further experiments on low-fidelity scenarios verify the flexibility.
Abstract (translated)
神经架构搜索（NAS）是一个计算上非常昂贵的优化问题，其搜索空间是离散、巨大且具有突变特性的。针对这一领域的一项关键研究努力集中在通过某些代理评估来加速NAS。与流行的基于预测的方法使用替代模型和超网络进行可微架构搜索不同，我们提出了一种优化代理OptiProxy-NAS，它将NAS作为一个端到端的优化框架进行了简化。特别是，利用一个代理表示，我们可以重新定义NAS空间使其成为连续、可微且平滑的空间。因此，任何基于梯度的方法都可以应用于松弛后的架构参数搜索。我们在计算机视觉、自然语言处理和资源受限的NAS三大领域中的12个NAS任务上进行了全面实验，证明了OptiProxy-NAS在搜索效果和效率上的优越性。此外，在低精度场景下的进一步实验验证了该方法的高度灵活性。
URL
https://arxiv.org/abs/2509.05656
