Abstract
Training time budget and size of the dataset are among the factors affecting the performance of a Deep Neural Network (DNN). This paper shows that Neural Architecture Search (NAS), Hyper Parameters Optimization (HPO), and Data Augmentation help DNNs perform much better while these two factors are limited. However, searching for an optimal architecture and the best hyperparameter values besides a good combination of data augmentation techniques under low resources requires many experiments. We present our approach to achieving such a goal in three steps: reducing training epoch time by compressing the model while maintaining the performance compared to the original model, preventing model overfitting when the dataset is small, and performing the hyperparameter tuning. We used NOMAD, which is a blackbox optimization software based on a derivative-free algorithm to do NAS and HPO. Our work achieved an accuracy of 86.0 % on a tiny subset of Mini-ImageNet at the ICLR 2021 Hardware Aware Efficient Training (HAET) Challenge and won second place in the competition. The competition results can be found at this http URL and our source code can be found at this http URL\_HAET2021.
Abstract (translated)
训练时间预算和数据集的大小是影响深度神经网络(DNN)性能的因素之一。本文表明,神经网络架构搜索(NAS)、超参数优化(HPO)和数据增强可以帮助DNN表现更好,尽管这两个因素受到限制。然而,在低资源环境下寻找最佳的架构和最佳超参数值,除了数据增强技术的良好组合,需要进行许多实验。我们提出了实现这一目标的三个步骤:通过压缩模型以减少训练迭代时间,保持与原始模型的性能相比的表现,在小数据集上防止模型过拟合,并进行超参数优化。我们使用了NOMAD,这是一个基于无后效性算法的黑盒优化软件,用于NAS和HPO。我们在ICLR 2021硬件Aware高效训练(HAET)挑战中实现了Mini-ImageNet小型子集的86.0%精度,并在竞赛中获得了第二名。竞赛结果可在这里找到,我们的源代码可在这里找到_HAET2021。
URL
https://arxiv.org/abs/2301.09264
