In a previous post on “What’s the deal with Neural Architecture Search?”, Liam Li and I discussed Neural Architecture Search (NAS) as a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. The article was essentially a primer on NAS in which we explained its connections to both AutoML and hyperparameter optimization. We concluded that while specialized NAS methods have demonstrated some promising empirical results, they are still not ready for widespread adoption.

Liam and I continued to explore the current state of NAS research and recently posted a paper “Random search and reproducibility for Neural Architecture Search” on arXiv. Our findings build upon the claims we made in our initial blog post, both in terms of the potential promise of NAS research and the current immaturity of the field.

In particular, we propose new NAS baselines to help ground the empirical results in this field. These baselines build off the facts that:

1. NAS is a specialized hyperparameter optimization problem.
2. Random search is a competitive baseline for hyperparameter optimization.

Remarkably, we demonstrate that our two simple random search baselines (one involving early stopping and the other relying on weight sharing) are competitive with (and in some cases outperform) existing algorithmically complex state-of-the-art NAS methods.

In the paper, we also explore the reproducibility issues of published NAS results, given the importance of reproducibility in production workloads. We note the lack of source material needed to reproduce NAS results exactly. For example, of the 12 papers published since 2018 at NeurIPS, ICML, and ICLR that introduce novel NAS methods, none of these papers provides adequate information (in the form of code, random seeds, and documentation) to exactly reproduce their reported empirical results. The paper also discusses the robustness of published results given the sources of variability in NAS experimental setups.

You can check out the details in our paper: https://arxiv.org/abs/1902.07638.

Have thoughts or questions about NAS? Let’s have a conversation.