Pro Yearly is on sale from $80 to $50! »

Automating Machine Learning

Automating Machine Learning

8ffe68e4b19092aab184e4aa09ca4bff?s=128

Andreas Mueller

July 15, 2016
Tweet

Transcript

  1. Andreas Mueller (NYU Center for Data Science, scikit-learn) Automatic Machine

    Learning?
  2. Why?

  3. Issues with current tools (scikit-learn)

  4. Flow chart / selecting model

  5. Selecting Hyper-Parameters

  6. Scikit-learn: Explicit is better than implicit make_pipeline( OneHotEncoder(), Imputer(), StandardScaler(),

    SVC())
  7. What? from automl import AutoClassifier clf = AutoClassifier().fit(X_train, y_train) >

    Current Accuracy: 70% (AUC .65) LinearSVC(C=1), 10sec > Current Accuracy: 76% (AUC .71) RandomForest(n_estimators=20) 30sec > Current Accuracy: 80% (AUC .74) RandomForest(n_estimators=500) 30sec
  8. Step 1: Automate Parameter Selection

  9. Step 2: Automate Model Selection

  10. Step 3: Automate Pipeline Selection

  11. How?

  12. Formalizing the Search Space Discrete and Continuous Parameters Conditional Parameters

    Fixed pipeline vs flexible pipeline
  13. Formalizing the Search Space Discrete and Continuous Parameters Conditional Parameters

    Fixed pipeline vs flexible pipeline
  14. Search Methods

  15. Exhaustive Search (Grid Search)

  16. Randomized Search

  17. Bayesian Optimization (SMBO)

  18. None
  19. None
  20. None
  21. Gaussian Processes

  22. Random Forest Based (SMAC)

  23. Non-parametric (TPE)

  24. None
  25. None
  26. Warm-starting and Meta-learning

  27. Meta-Learning optimization Algorithm + Parameters Dataset 1

  28. Meta-Learning optimization Algorithm + Parameters Dataset 3 optimization Algorithm +

    Parameters Dataset 2 optimization Algorithm + Parameters Dataset 1
  29. Meta-Learning Meta-Features 1 optimization Algorithm + Parameters Dataset 3 optimization

    Algorithm + Parameters Dataset 2 optimization Algorithm + Parameters Dataset 1 Meta-Features 2 Meta-Features 3 ML model
  30. Meta-Learning Meta-Features 1 optimization Algorithm + Parameters Dataset 3 optimization

    Algorithm + Parameters Dataset 2 optimization Algorithm + Parameters Dataset 1 Meta-Features 2 Meta-Features 3 ML model New Dataset ML model Algorithm + Parameters
  31. Meta-Features

  32. Existing Approaches

  33. auto-sklearn (Hutter, Feurer, Eggensperger) http://automl.github.io/auto-sklearn/stable/

  34. Autoweka

  35. Hyperopt-sklearn

  36. TPot

  37. Spearmint https://github.com/HIPS/Spearmint

  38. Scikit-optimize

  39. Within Scikit-learn • GridSearchCV • RandomizedSearchCV • BayesianSearchCV (coming) •

    Searching over Pipelines (coming) • Built-in parameter ranges (coming)
  40. TODO Clean separation of: • Model Search Space • Pipeline

    Search Space • Optimization Method • Meta-Learning • Exploit prior knowledge better! • Usability • Runtime consideration
  41. TODO Clean separation of: • Model Search Space • Pipeline

    Search Space • Optimization Method • Meta-Learning • Exploit prior knowledge better! • Usability • Runtime consideration • Data subsampling
  42. Criticism

  43. Randomized Search works well

  44. Do we need 100 Classifiers? Do we need Complex pipelines?

  45. I don’t want a black-box!

  46. 46 http://oreilly.com/pub/get/scipy

  47. 47 Material • Random Search for Hyper-Parameter Optimization (Bergstra, Bengio)

    • Efficient and Robust Automated Machine Learning (Feurer et al) [autosklearn] • http://automl.github.io/auto-sklearn/stable/ • Efficient Hyperparameter Optimization and Infinitely Many Armed Bandits (Lie et. al) [hyperband] https://arxiv.org/abs/1603.06560 • Scalable Bayesian Optimization Using Deep Neural Networks [Snoek et al]
  48. 48 @amuellerml @amueller amueller@nyu.edu http://amueller.io Thank you.