%BJDIJ
)PSJUB %"354
%JGGFSFOUJBCMF
"SDIJUFDUVSF
4FBSDI )BOYJBP
-JV
,BSFO
4JNPOZBO
:JNJOH
:BOH
*$-3  

$POUFOUT w *OUSPEVDUJPO
PG
/FVSBM
"SDIJUFDUVSF
4FBSDI
w 1SPQPTFE
.FUIPE
w &YQFSJNFOUT
w $PODMVTJPO  

Introduction of Neural Architecture Search(1/2) What is Neural Architecture Search? • Neural networks and hyper-parameters are still hard to design • Related works • Hyper-parameter optimization[Saxena+, NIPS16], [Bergstra+, JMLR12] • Not architecture-level • Evolution algorithms[Stanley+, Artificial Life 09], [Floreano+, EI08] • Less practical at a large scale Need to construct general framework automatically!  

Introduction of Neural Architecture Search(2/2) Neural Architecture Search with Reinforcement Learning[Zoph+ ICLR17] • Proposed reinforcement learning based method with a RNN controller • Not differentiable! RNN controller Heavy heavy computational costs!  

a set of operations, : a weight of a node between i and j, apply operation O : α(i,j) o o(x) : • Objectives Cannot compute… Proposed method(1/) Define Differentiable NAS 

 where Hessian matrix… So computation is heavy If hyper-parameter = 0, then no need to compute! (Discuss later about a classification accuracy) Proposed method(2/) Define Differentiable NAS 

 where Approximate a finite difference O(αw) O(α + w) Proposed method(3/) Define Differentiable NAS 

Proposed method(4/) Algorithm • Alternately update and Lval Ltrain 

Experiments(1/) CIFAR-10 classification First order means the hyper-parameter is 0 

Experiments(2/) ImageNet classification Fastest! 

Conclusion • Achieve differentiable NAS. • Achieve the fastest time for training and searching. • Available my implementation from https://github.com/UdonDa/DARTS_pytorch • (Only CIFAR-10)