Abstract
Evaluation is a critical but costly procedure in neural architecture search (NAS). Performance predictors have been widely adopted to reduce evaluation costs by directly estimating architecture performance. The effectiveness of predictors is heavily influenced by the choice of loss functions. While traditional predictors employ regression loss functions to evaluate the absolute accuracy of architectures, recent approaches have explored various ranking-based loss functions, such as pairwise and listwise ranking losses, to focus on the ranking of architecture performance. Despite their success in NAS, the effectiveness and characteristics of these loss functions have not been thoroughly investigated. In this paper, we conduct the first comprehensive study on loss functions in performance predictors, categorizing them into three main types: regression, ranking, and weighted loss functions. Specifically, we assess eight loss functions using a range of NAS-relevant metrics on 13 tasks across five search spaces. Our results reveal that specific categories of loss functions can be effectively combined to enhance predictor-based NAS. Furthermore, our findings could provide practical guidance for selecting appropriate loss functions for various tasks. We hope this work provides meaningful insights to guide the development of loss functions for predictor-based methods in the NAS community.
Abstract (translated)
评估是神经架构搜索（NAS）中的一个关键但成本高昂的步骤。性能预测器被广泛采用，以通过直接估算架构性能来减少评估成本。预测器的有效性很大程度上取决于损失函数的选择。传统预测器使用回归损失函数来评估架构的绝对准确性，而近期方法则探索了各种基于排名的损失函数，如成对和列表排序损失，重点关注架构性能的排名。尽管这些方法在NAS中取得了成功，但它们的效果和特性尚未得到彻底研究。 在这篇论文中，我们进行了首个关于性能预测器中的损失函数的全面研究，将它们分类为三大类：回归、排名以及加权损失函数。具体而言，我们在五个搜索空间内的13个任务上使用一系列NAS相关指标评估了八个不同的损失函数。我们的结果表明，特定类型的损失函数可以有效结合以增强基于预测器的NAS方法。此外，我们的发现可为不同任务选择适当的损失函数提供实用指导。 我们希望这项工作能为NAS社区开发用于基于预测的方法的损失函数提供有意义的见解和指导。
URL
https://arxiv.org/abs/2506.05869
