Abstract
We propose a neural architecture search (NAS) algorithm, Petridish, to iteratively add shortcut connections to existing network layers. The added shortcut connections effectively perform gradient boosting on the augmented layers. The proposed algorithm is motivated by the feature selection algorithm forward stage-wise linear regression, since we consider NAS as a generalization of feature selection for regression, where NAS selects shortcuts among layers instead of selecting features. In order to reduce the number of trials of possible connection combinations, we train jointly all possible connections at each stage of growth while leveraging feature selection techniques to choose a subset of them. We experimentally show this process to be an efficient forward architecture search algorithm that can find competitive models using few GPU days in both the search space of repeatable network modules (cell-search) and the space of general networks (macro-search). Petridish is particularly well-suited for warm-starting from existing models crucial for lifelong-learning scenarios.
Abstract (translated)
我们提出了一种神经架构搜索（NAS）算法Petridish，用于迭代地向现有网络层添加快捷连接。添加的快捷方式连接有效地在增强的层上执行渐变增强。提出的算法是由特征选择算法向前阶段线性回归驱动的，因为我们把nas看作是回归特征选择的一种推广，nas选择层间的快捷方式而不是选择特征。为了减少可能的连接组合的测试次数，我们在每个增长阶段联合训练所有可能的连接，同时利用特性选择技术来选择其中的一个子集。实验表明，该算法是一种高效的前向结构搜索算法，在可重复网络模块（单元搜索）的搜索空间和一般网络（宏搜索）的搜索空间中，都可以用几天的GPU时间找到竞争模型。Petridish特别适合从对终身学习场景至关重要的现有模型进行热启动。
URL
https://arxiv.org/abs/1905.13360
