Automating Machine Learning

Transcript

1. Andreas Mueller
   (NYU Center for Data Science, scikit-learn)
   Automatic
   Machine Learning?
   

   View Slide

2. Why?
   

   View Slide

3. Issues with current tools (scikit-learn)
   

   View Slide

4. Flow chart / selecting model
   

   View Slide

5. Selecting Hyper-Parameters
   

   View Slide

6. Scikit-learn: Explicit is better than implicit
   make_pipeline(
   OneHotEncoder(),
   Imputer(),
   StandardScaler(),
   SVC())
   

   View Slide

7. What?
   from automl import AutoClassifier
   clf = AutoClassifier().fit(X_train, y_train)
   > Current Accuracy: 70% (AUC .65) LinearSVC(C=1), 10sec
   > Current Accuracy: 76% (AUC .71) RandomForest(n_estimators=20) 30sec
   > Current Accuracy: 80% (AUC .74) RandomForest(n_estimators=500) 30sec
   

   View Slide

8. Step 1: Automate Parameter Selection
   

   View Slide

9. Step 2: Automate Model Selection
   

   View Slide

10. Step 3: Automate Pipeline Selection
    

    View Slide

11. How?
    

    View Slide

12. Formalizing the Search Space
    Discrete and Continuous Parameters
    Conditional Parameters
    Fixed pipeline vs flexible pipeline
    

    View Slide

13. Formalizing the Search Space
    Discrete and Continuous Parameters
    Conditional Parameters
    Fixed pipeline vs flexible pipeline
    

    View Slide

14. Search Methods
    

    View Slide

15. Exhaustive Search (Grid Search)
    

    View Slide

16. Randomized Search
    

    View Slide

17. Bayesian Optimization (SMBO)
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. Gaussian Processes
    

    View Slide

22. Random Forest Based (SMAC)
    

    View Slide

23. Non-parametric (TPE)
    

    View Slide

24. View Slide

25. View Slide

26. Warm-starting and Meta-learning
    

    View Slide

27. Meta-Learning
    optimization Algorithm + Parameters
    Dataset 1
    

    View Slide

28. Meta-Learning
    optimization Algorithm + Parameters
    Dataset 3 optimization Algorithm + Parameters
    Dataset 2 optimization Algorithm + Parameters
    Dataset 1
    

    View Slide

29. Meta-Learning
    Meta-Features 1
    optimization Algorithm + Parameters
    Dataset 3 optimization Algorithm + Parameters
    Dataset 2 optimization Algorithm + Parameters
    Dataset 1
    Meta-Features 2
    Meta-Features 3
    ML model
    

    View Slide

30. Meta-Learning
    Meta-Features 1
    optimization Algorithm + Parameters
    Dataset 3 optimization Algorithm + Parameters
    Dataset 2 optimization Algorithm + Parameters
    Dataset 1
    Meta-Features 2
    Meta-Features 3
    ML model
    New Dataset ML model Algorithm + Parameters
    

    View Slide

31. Meta-Features
    

    View Slide

32. Existing Approaches
    

    View Slide

33. auto-sklearn
    (Hutter, Feurer, Eggensperger)
    http://automl.github.io/auto-sklearn/stable/
    

    View Slide

34. Autoweka
    

    View Slide

35. Hyperopt-sklearn
    

    View Slide

36. TPot
    

    View Slide

37. Spearmint
    https://github.com/HIPS/Spearmint
    

    View Slide

38. Scikit-optimize
    

    View Slide

39. Within Scikit-learn
    ●
    GridSearchCV
    ●
    RandomizedSearchCV
    ●
    BayesianSearchCV (coming)
    ●
    Searching over Pipelines (coming)
    ●
    Built-in parameter ranges (coming)
    

    View Slide

40. TODO
    Clean separation of:
    ●
    Model Search Space
    ●
    Pipeline Search Space
    ●
    Optimization Method
    ●
    Meta-Learning
    ●
    Exploit prior knowledge better!
    ●
    Usability
    ●
    Runtime consideration
    

    View Slide

41. TODO
    Clean separation of:
    ●
    Model Search Space
    ●
    Pipeline Search Space
    ●
    Optimization Method
    ●
    Meta-Learning
    ●
    Exploit prior knowledge better!
    ●
    Usability
    ●
    Runtime consideration
    ●
    Data subsampling
    

    View Slide

42. Criticism
    

    View Slide

43. Randomized Search works well
    

    View Slide

44. Do we need 100 Classifiers?
    Do we need Complex pipelines?
    

    View Slide

45. I don’t want a black-box!
    

    View Slide

46. 46
    http://oreilly.com/pub/get/scipy
    

    View Slide

47. 47
    Material
    ●
    Random Search for Hyper-Parameter Optimization (Bergstra, Bengio)
    ●
    Efficient and Robust Automated Machine Learning (Feurer et al)
    [autosklearn]
    ●
    http://automl.github.io/auto-sklearn/stable/
    ●
    Efficient Hyperparameter Optimization and Infinitely Many Armed
    Bandits (Lie et. al) [hyperband] https://arxiv.org/abs/1603.06560
    ●
    Scalable Bayesian Optimization Using Deep Neural Networks [Snoek
    et al]
    

    View Slide

48. 48
    @amuellerml
    @amueller
    amueller@nyu.edu
    http://amueller.io
    Thank you.
    

    View Slide