Abstract
Differentiable Architecture Search (DARTS) is an efficient Neural Architecture Search (NAS) method but suffers from robustness, generalization, and discrepancy issues. Many efforts have been made towards the performance collapse issue caused by skip dominance with various regularization techniques towards operation weights, path weights, noise injection, and super-network redesign. It had become questionable at a certain point if there could exist a better and more elegant way to retract the search to its intended goal -- NAS is a selection problem. In this paper, we undertake a simple but effective approach, named Smooth Activation DARTS (SA-DARTS), to overcome skip dominance and discretization discrepancy challenges. By leveraging a smooth activation function on architecture weights as an auxiliary loss, our SA-DARTS mitigates the unfair advantage of weight-free operations, converging to fanned-out architecture weight values, and can recover the search process from skip-dominance initialization. Through theoretical and empirical analysis, we demonstrate that the SA-DARTS can yield new state-of-the-art (SOTA) results on NAS-Bench-201, classification, and super-resolution. Further, we show that SA-DARTS can help improve the performance of SOTA models with fewer parameters, such as Information Multi-distillation Network on the super-resolution task.
Abstract (translated)
不同可微架构搜索（DARTS）是一种高效的神经网络架构搜索（NAS）方法，但在鲁棒性、泛化能力和差异性方面存在问题。针对由跳跃连接主导导致的性能崩溃问题，已经采取了多种正则化技术来改善操作权重、路径权重、噪声注入和超级网络的设计。在某个阶段，人们开始质疑是否存在一种更好的且更优雅的方法可以使搜索回归到它的目标——即NAS本质上是一个选择问题。在这篇论文中，我们采用了一种简单但有效的方法，命名为平滑激活DARTS（SA-DARTS），以克服跳跃连接主导和离散化差异的挑战。通过在架构权重上应用一个平滑激活函数作为辅助损失，我们的SA-DARTS减轻了无权操作不公正的优势，收敛到扇出式的架构权重值，并能从跳跃主导初始化中恢复搜索过程。通过理论分析和实验验证，我们证明了SA-DARTS可以在NAS-Bench-201、分类任务以及超分辨率上产生新的最先进的（SOTA）结果。此外，我们展示了SA-DARTS可以帮助提升具有较少参数的最新模型在特定任务上的性能，例如信息多重蒸馏网络在超分辨率任务中的应用。
URL
https://arxiv.org/abs/2504.16306
