A Modern Guide to Hyperparameter Optimization Richard Liaw

2 ©2017 RISELab Deep learning is taking over the world

3 ©2017 RISELab def train_model(): model = ConvNet() optimizer = Optimizer() for batch in Dataset(): loss, acc = model.train(batch) optimizer.update(model, loss)

4 ©2017 RISELab def train_model(): model = ConvNet() optimizer = Optimizer() for batch in Dataset(): loss, acc = model.train(batch) optimizer.update(model, loss)

5 ©2017 RISELab def train_model(): model = ConvNet(layers, activations, drop... optimizer = Optimizer(lr, momentum, decay...) for batch in Dataset(standardize, shift, ...) loss, acc = model.train(batch) optimizer.update(model, loss) Tune this!

6 ©2017 RISELab Hyperparameters matter!

7 ©2017 RISELab Goal of hyperparameter tuning Maximize Model Performance Minimize Time spent Minimize money spent

8 ©2017 RISELab Overview of hyperparameter tuning techniques Grid Search Bayesian Optimization HyperBand (Bandits) Population- based Training Random Search Definition: “trial” = “one configuration evaluation”

9 ©2017 RISELab Grid Search tl;dr - Cross-product of all possible configurations. for rate in [0.1, 0.01, 0.001]: for hidden_layers in [2, 3, 4]: for param in ["a", "b", "c"]: train_model( rate, hidden_layers, param) Benefit: 1. Explainable 2. Easily parallelizable Problems: Inefficient/expensive ⇐ 27 evaluations!

10 ©2017 RISELab Grid Search Random Search

11 ©2017 RISELab Random Search tldr: Sample configurations. for i in num_samples: train_model( rate=sample(0.001, 0.1), hidden_layers=sample(2, 4), param=sample(["a", "b", "c"])) Benefit: 1. Better coverage on important parameters 2. Easily parallelizable 3. Hard to beat on high dimensions Problems: Still inefficient/expensive!

12 ©2017 RISELab What if we used some prior information to guide our tuning process? photo from github.com/ fmfn/bayesianoptimization

13 ©2017 RISELab Bayesian Optimization opt = Optimizer( lr=(0.01, 0.1), layers=(2, 5) ) for i in range(9): config = opt.ask() score = train_model(config) opt.tell(config, score) Model-based optimization of hyperparameters. Libraries: Hyperopt, Scikit-optimize Benefit: 1. Can utilize prior information 2. Semi-Parallelizable (Kandasamy 2018) Still can do better!

14 ©2017 RISELab We can do better by exploiting structure! Why waste resources on this?

15 ©2017 RISELab HyperBand/ASHA (early stopping algorithms) trial = sample_from(hyperparameter_space) while trial.iter < max_epochs: trial.run_one_epoch() if trial.at_cutoff(): if is_top_fraction(trial, trial.iter): trial.extend_cutoff() else: # allow new trials to start trial.pause(); break Intuition: 1. Compare relative performance 2. Terminate bad performing trials 3. Continue better trials for longer period of time Notes: 1. Can be combined with Bayesian Optimization 2. Can be easily parallelized

16 ©2017 RISELab But what about dynamic hyperparameters? Changed Learning rate!

17 ©2017 RISELab Population-based training 0.1 0.2 0.3 0.4 0.1 0.2 0.3 0.1 0.2 0.3 .15 0.1 0.3 .15 Main idea: Evaluate a population in parallel. Terminate lowest performers. Copy weights of the best performers and mutates hyperparameters 0.4 Benefits: 1. Easily parallelizable 2. Can search over “schedules” 3. Terminates bad performers

18 ©2017 RISELab Does it really work?

19 ©2017 RISELab OK, but there’s no way I’m going to implement all of these algorithms...

A library for distributed hyperparameter search tune.io

21 ©2017 RISELab Tune and many others!

22 ©2017 RISELab Tune handles hyperparameter search execution. tune.io

23 ©2017 RISELab Resource Aware Scheduling Framework Agnostic Tune is built with Deep Learning as a priority.

24 ©2017 RISELab ● HyperOpt (TPE) ● Bayesian Optimization ● SigOpt ● Nevergrad ● Scikit-Optimize ● Ax/Botorch (PyTorch BayesOpt) Tune Algorithm Offerings Search Algorithms Provided ● Population-based Training ● HyperBand ● ASHA ● Median-stopping Rule ● BOHB Trial Schedulers Provided

25 ©2017 RISELab Tune: Powers Many Open Source Projects

26 ©2017 RISELab Native Integration with TensorBoard HParams

27 ©2017 RISELab Resources PyData Demo: https://github.com/richardliaw/pydata_demo Tune Documentation: http://tune.io Tune Tutorial: https://github.com/ray-project/tutorial/

live demo

29 ©2017 RISELab ray.readthedocs.io/en/latest/tune.html ray.readthedocs.io/en/latest/tune.html def train_model(config={}): model = ConvNet(config) for i in range(steps): loss, acc = model.train()

30 ©2017 RISELab ray.readthedocs.io/en/latest/tune.html ray.readthedocs.io/en/latest/tune.html from ray.tune import run, track def train_model(config={}): model = ConvNet(config) for i in range(steps): loss, acc = model.train() track.log(mean_loss=loss)

31 ©2017 RISELab ray.readthedocs.io/en/latest/tune.html ray.readthedocs.io/en/latest/tune.html train_model(config={“learning_rate": 0.1}) def train_model(config={}): model = ConvNet(config) for i in range(steps): loss, acc = model.train() track.log(mean_loss=loss)

32 ©2017 RISELab ray.readthedocs.io/en/latest/tune.html ray.readthedocs.io/en/latest/tune.html def train_model(config={}): model = ConvNet(config) for i in range(steps): loss, acc = model.train() track.log(mean_loss=loss) tune.run(train_model, config={“learning_rate”: 0.1})

33 ©2017 RISELab ray.readthedocs.io/en/latest/tune.html ray.readthedocs.io/en/latest/tune.html tune.run(train_model, config={“learning_rate”: 0.1}, num_samples=100) def train_model(config): model = ConvNet(config) for i in range(steps): loss, acc = model.train() track.log(mean_loss=loss)

34 ©2017 RISELab ray.readthedocs.io/en/latest/tune.html ray.readthedocs.io/en/latest/tune.html tune.run(train_model, config={“learning_rate”: 0.1}, num_samples=100, upload_dir="s3://my_bucket") def train_model(config): model = ConvNet(config) for i in range(steps): loss, acc = model.train() track.log(mean_loss=loss)

35 ©2017 RISELab ray.readthedocs.io/en/latest/tune.html ray.readthedocs.io/en/latest/tune.html tune.run(train_model, config={ “learning_rate”: tune.uniform(0.001, 0.1)}, num_samples=100, upload_dir="s3://my_bucket", scheduler=AsyncHyperBandScheduler()) def train_model(config): model = ConvNet(config) for i in range(steps): loss, acc = model.train() track.log(mean_loss=loss)