Abstract
Pareto front profiling in multi-objective optimization (MOO), i.e. finding a diverse set of Pareto optimal solutions, is challenging, especially with expensive objectives like neural network training. Typically, in MOO neural architecture search (NAS), we aim to balance performance and hardware metrics across devices. Prior NAS approaches simplify this task by incorporating hardware constraints into the objective function, but profiling the Pareto front necessitates a search for each constraint. In this work, we propose a novel NAS algorithm that encodes user preferences for the trade-off between performance and hardware metrics, and yields representative and diverse architectures across multiple devices in just one search run. To this end, we parameterize the joint architectural distribution across devices and multiple objectives via a hypernetwork that can be conditioned on hardware features and preference vectors, enabling zero-shot transferability to new devices. Extensive experiments with up to 19 hardware devices and 3 objectives showcase the effectiveness and scalability of our method. Finally, we show that, without additional costs, our method outperforms existing MOO NAS methods across qualitatively different search spaces and datasets, including MobileNetV3 on ImageNet-1k and a Transformer space on machine translation.
Abstract (translated)
在多目标优化（MOO）中进行帕累托前沿分析（PFA），即寻找多样性的帕累托最优解决方案，是一个具有挑战性的任务，尤其是在具有昂贵目标（如神经网络训练）的情况下。通常，在MOO神经架构搜索（NAS）中，我们试图在设备之间平衡性能和硬件指标。以前NAS方法通过将硬件约束融入目标函数来简化这一任务，但PFA需要对每个约束进行搜索。在这项工作中，我们提出了一个新颖的NAS算法，它将用户对性能和硬件指标之间的权衡的偏好编码到用户偏好的联合架构分布中，并仅在一次搜索运行中生成具有代表性的多样性的架构。为此，我们通过一个可以条件化硬件特征和偏好向量的超网络，对设备之间的联合 architectural 分布进行参数化，实现零 shots 传输到新设备。我们对多达19个硬件设备和3个目标进行了广泛的实验，展示了我们方法的有效性和可扩展性。最后，我们证明了，在没有额外费用的情况下，我们的方法在定性不同的搜索空间和数据集上优于现有的MOO NAS方法，包括在ImageNet-1k上使用移动NetV3和机器翻译空间上使用Transformer。
URL
https://arxiv.org/abs/2402.18213
