Learning scikit-learn -- An Introduction to Machine Learning in Python @ PyData Chicago 2016

Learning scikit-learn -- An Introduction to Machine Learning in Python @ PyData Chicago 2016

This tutorial will teach you the basics of scikit-learn. I will give you a brief overview of the basic concepts of classification and regression analysis, how to build powerful predictive models from labeled data. The accompanying GitHub repository can be found at https://github.com/rasbt/pydata-chicago2016-ml-tutorial.

3324b5ee3a1f4060057dad7c215265f5?s=128

Sebastian Raschka

August 24, 2016
Tweet

Transcript

  1. Sebastian Raschka! PyData Chicago 2016! Chicago, The University of Illinois

    • August 26, 2016! Learning – An Introduction to Machine Learning in Python
  2. Sebastian Raschka, Learning scikit-learn! Links & Info! Contact:! o  E-mail:

    mail@sebastianraschka.com! o  Website: http://sebastianraschka.com! o  Twitter: @rasbt! o  GitHub: rasbt! Tutorial Material on GitHub:! https://github.com/rasbt/pydata-chicago2016-ml-tutorial! 2!
  3. Sebastian Raschka, Learning scikit-learn! Let’s Not Stress! ! 3! This

    is an introductory tutorial, and we are here to learn! ! Please ask questions!!
  4. Sebastian Raschka, Learning scikit-learn! What can Machine Learning do for

    us?! 4! https://commons.wikimedia.org/wiki/ File:Google_self_driving_car_at_the_Googleplex.jpg! Photo by Michasel Shick, CC BY-SA 4.0 Iit! h"ps://flic.kr/p/5BLW6G [CC BY 2.0]
  5. Sebastian Raschka, Learning scikit-learn! What is Machine Learning?! 5! Outputs

    ! (labels)! Inputs ! (observations)! Computer! Program! Spam/Non-Spam! Labels! Emails! Classification Algorithm! Spam Filter!
  6. Sebastian Raschka, Learning scikit-learn! 3 Types of Learning! 6! Reinforcement!

    Supervised! Unsupervised!
  7. Sebastian Raschka, Learning scikit-learn! Working with Labeled Data! 7! Supervised

    Learning! ?! x (“input”) y (“output”) x1 (“input”) x2 (“input”) ?! Regression Classification
  8. Sebastian Raschka, Learning scikit-learn! Working with Unlabeled Data! 8! Unsupervised

    Learning! Clustering Compression
  9. Sebastian Raschka, Learning scikit-learn! Topics! 9! 1.  Introduction to Machine

    Learning! 2.  Linear Regression 3.  Introduction to Classification! 4.  Feature Preprocessing & scikit-learn Pipelines! 5.  Dimensionality Reduction: Feature Selection & Extraction! 6.  Model Evaluation & Hyperparameter Tuning!
  10. Sebastian Raschka, Learning scikit-learn! Simple Linear Regression! 10! y (response

    variable) x (explanatory variable) (xi , yi ) ŷ = w0 + w1 x w0 (intercept) w1 (slope) = Δy / Δx Δx Δy verMcal offset |ŷ − y|
  11. Sebastian Raschka, Learning scikit-learn! 11! features (columns)! x0 x1 …

    xm x0,0! x0,1! x1,0! x1,1! x2,0! x2,1! x3,0! x3,1! .! .! .! xn,0! xn,1! …! xn,m! X=! y0! y1! y2! y3! .! .! .! yn! y=! Data Representation! samples (rows)!
  12. Sebastian Raschka, Learning scikit-learn! Learning Algorithm Hyperparameter Values Model Prediction

    Test Labels Performance Model Learning Algorithm Hyperparameter Values Final Model 2 3 4 1 Test Labels Test Data Training Data Training Labels Data Labels Data Labels Training Data Training Labels Test Data “Basic” Supervised Learning Workflow! 12!
  13. Sebastian Raschka, Learning scikit-learn! Code Examples! 13! Jupyter Notebook!

  14. Sebastian Raschka, Learning scikit-learn! Topics! 14! 1.  Introduction to Machine

    Learning! 2.  Linear Regression! 3.  Introduction to Classification 4.  Feature Preprocessing & scikit-learn Pipelines! 5.  Dimensionality Reduction: Feature Selection & Extraction! 6.  Model Evaluation & Hyperparameter Tuning!
  15. Sebastian Raschka, Learning scikit-learn! Scikit-learn API! 15! class SupervisedEstimator(...): def

    __init__(self, hyperparam, ...): ... def fit(self, X, y): ... return self def predict(self, X): ... return y_pred def score(self, X, y): ... return score ...
  16. Sebastian Raschka, Learning scikit-learn! Iris Dataset! 16! Iris-Versicolor! Iris-Setosa! Iris-Setosa!

  17. Sebastian Raschka, Learning scikit-learn! 17! features (columns)! sepal length [cm]

    ! sepal width [cm] ! petal length [cm] ! petal width [cm] ! 1! 5.1 ! 3.5 ! 1.4! 0.2! 2! 4.9 ! 3.0 ! 1.4! 0.2! 50! 6.4 ! 3.5 ! 4.5! 1.2! .! .! .! 150! 5.9 ! 3.0 ! 5.0! 1.8 ! X=! setosa ! setosa ! versicolor ! .! .! .! virginica ! y=! Iris Dataset! samples (rows)! sepal ! petal !
  18. Sebastian Raschka, Learning scikit-learn! Note about Non-Stratified Splits! 18! § 

    training set  38 x Setosa, 28 x Versicolor, 34 x Virginica! §  test set  12 x Setosa, 22 x Versicolor, 16 x Virginica!
  19. Sebastian Raschka, Learning scikit-learn! 19! Linear Regression Recap! Σ .!

    .! .! w1! wm! w2! w0! x1! 1 ! x2! xm! !y! Activation function ! Net input function! a! z! Unit step function! Predicted output! Weight coefficients! Input values! Bias unit!
  20. Sebastian Raschka, Learning scikit-learn! Logistic Regression, a Generalized Linear Model!

    20! Σ .! .! .! w1! wm! w2! w0! x1! 1 ! x2! xm! !y! Activation function ! Net input function! a! z! Unit step function! Predicted class label! Weight coefficients! Input values! Bias unit! Predicted probability!
  21. Sebastian Raschka, Learning scikit-learn! A “Lazy Learner:” K-Nearest Neighbors Classifier!

    21! x1! ?! 3 ×! 1 ×! 1 ×! Predict ! ?! = ! x2!
  22. Sebastian Raschka, Learning scikit-learn! Code Examples! 22! Jupyter Notebook!

  23. Sebastian Raschka, Learning scikit-learn! Topics! 23! 1.  Introduction to Machine

    Learning! 2.  Linear Regression! 3.  Introduction to Classification! 4.  Feature Preprocessing & scikit-learn Pipelines 5.  Dimensionality Reduction: Feature Selection & Extraction! 6.  Model Evaluation & Hyperparameter Tuning!
  24. Sebastian Raschka, Learning scikit-learn! Categorical Variables! 24! color! size! price!

    class label! red! M! $10.49! 0! blue! XL! $15.00! 1! green! L! $12.99! 1!
  25. Sebastian Raschka, Learning scikit-learn! Encoding Categorical Variables! 25! color! size!

    price! class label! red! M! $10.49! 0! blue! XL! $15.00! 1! green! L! $12.99! 1! size! 0! 2! 1! red! blue! green! 1! 0! 0! 0! 1! 0! 0! 0! 1!
  26. Sebastian Raschka, Learning scikit-learn! Feature Normalization! 26! feature! minmax! z-score!

    1.0! 0.0! -1.46385! 2.0! 0.2! -0.87831! 3.0! 0.4! -0.29277! 4.0! 0.6! 0.29277! 5.0! 0.8! 0.87831! 6.0! 1.0! 1.46385! Min-max scaling! Z-score standardization!
  27. Sebastian Raschka, Learning scikit-learn! Scikit-learn API! 27! class UnsupervisedEstimator(...): def

    __init__(self, ...): ... def fit(self, X): ... return self def transform(self, X): ... return X_transf def predict(self, X): ... return pred
  28. Sebastian Raschka, Learning scikit-learn! Scikit-learn Pipelines! 28! Class labels! Training

    data! Test data! Learning Algorithm! Dimensionality Reduction! Scaling! Model! Pipeline! fit fit & transform fit & transform fit transform transform Class labels! predict predict
  29. Sebastian Raschka, Learning scikit-learn! Code Examples! 29! Jupyter Notebook!

  30. Sebastian Raschka, Learning scikit-learn! Topics! 30! 1.  Introduction to Machine

    Learning! 2.  Linear Regression! 3.  Introduction to Classification! 4.  Feature Preprocessing & scikit-learn Pipelines! 5.  Dimensionality Reduction: Feature Selection & Extraction 6.  Model Evaluation & Hyperparameter Tuning!
  31. Sebastian Raschka, Learning scikit-learn! Dimensionality Reduction – why?! 31! [cm]

    [cm] [cm] [cm] [cm] [cm] [cm] [cm]
  32. Sebastian Raschka, Learning scikit-learn! Dimensionality Reduction – why?! 32! predictive

    performance! predictive performance! storage & speed! visualization & interpretability!
  33. Sebastian Raschka, Learning scikit-learn! Recursive Feature Elimination! 33! available features:!

    [ w1 w2 w3 w4 ] [ w1 w2 w4 ] [ w1 w4 ] [ w4 ] [ f1 f2 f3 f4 ] fit model, remove lowest weight, repeat! fit model, remove lowest weight, repeat! fit model, remove lowest weight, repeat!
  34. Sebastian Raschka, Learning scikit-learn! Sequential Feature Selection! 34! [ f1

    f2 f3 f4 ] [ f1 ] [ f2 ] [ f3 ] [ f4 ] [ f3 f1 ] [ f3 f2 ] [ f3 f4 ] [ f3 f2 f1 ] [ f3 f2 f4 ] available features:! fit model, pick best, repeat! fit model, pick best, repeat!
  35. Sebastian Raschka, Learning scikit-learn! Principal Component Analysis! 35! x1! x2!

    PC1! PC2!
  36. Sebastian Raschka, Learning scikit-learn! Code Examples! 36! Jupyter Notebook!

  37. Sebastian Raschka, Learning scikit-learn! Topics! 37! 1.  Introduction to Machine

    Learning! 2.  Linear Regression! 3.  Introduction to Classification! 4.  Feature Preprocessing & scikit-learn Pipelines! 5.  Dimensionality Reduction: Feature Selection & Extraction! 6.  Model Evaluation & Hyperparameter Tuning
  38. Sebastian Raschka, Learning scikit-learn! Learning Algorithm Hyperparameter Values Model Prediction

    Test Labels Performance Model Learning Algorithm Hyperparameter Values Final Model 2 3 4 1 Test Labels Test Data Training Data Training Labels Data Labels Data Labels Training Data Training Labels Test Data “Basic” Supervised Learning Workflow! 38!
  39. Sebastian Raschka, Learning scikit-learn! Bias and Variance! 39!

  40. Sebastian Raschka, Learning scikit-learn! Learning Curves! 40!

  41. Sebastian Raschka, Learning scikit-learn! Repeated Holdout! 41!

  42. Sebastian Raschka, Learning scikit-learn! Holdout and Hyperparameter Tuning I! 42!

    2 1 Data Labels Training Data Validation Data Validation Labels Test Data Test Labels Training Labels Performance Model Validation Data Validation Labels Prediction Performance Model Validation Data Validation Labels Prediction Performance Model Validation Data Validation Labels Prediction Best Model Learning Algorithm Hyperparameter values Model Hyperparameter values Hyperparameter values Model Model Training Data Training Labels 3 Best Hyperparameter values
  43. Sebastian Raschka, Learning scikit-learn! Holdout and Hyperparameter Tuning II! 43!

    Learning Algorithm Best Hyperparameter Values Final Model 6 Data Labels Prediction Test Labels Performance Model 4 Test Data Learning Algorithm Best Hyperparameter Values Model Training Data Training Labels 5 Validation Data Validation Labels
  44. Sebastian Raschka, Learning scikit-learn! 1st 2nd 3rd 4th 5th K

    Iterations (K-Folds) Validation Fold Training Fold Learning Algorithm Hyperparameter Values Model Training Fold Data Training Fold Labels Prediction Performance Model Validation Fold Data Validation Fold Labels Performance Performance Performance Performance Performance 1 2 3 4 5 Performance 1 10 ∑ 10 i=1 Performancei = This work by Sebastian Raschka is licensed under a Creative Commons Attribution 4.0 International License. K-fold Cross-Validation! 44!
  45. Sebastian Raschka, Learning scikit-learn! 45! K-fold Cross-Validation Workflow I! Test

    Labels Test Data Training Data Training Labels Data Labels Model Model Model Learning Algorithm Hyperparameter values Hyperparameter values Hyperparameter values Training Data Training Labels Learning Algorithm Best Hyperparameter Values Model Training Data Training Labels 2 1 3
  46. Sebastian Raschka, Learning scikit-learn! 46! K-fold Cross-Validation Workflow II! Learning

    Algorithm Best Hyperparameter Values Model Training Data Training Labels Prediction Test Labels Performance Model Test Data Learning Algorithm Best Hyperparameter Values Final Model Data Labels 3 4 5
  47. Sebastian Raschka, Learning scikit-learn! Code Examples! 47! Jupyter Notebook!

  48. Sebastian Raschka, Learning scikit-learn! Performance Metrics! 48! http://scikit-learn.org/stable/modules/model_evaluation.html! !

  49. Sebastian Raschka, Learning scikit-learn! Further Resources! 50! Documentation:! http://scikit-learn.org! Mailing

    list:! https://mail.python.org/mailman/listinfo/scikit-learn!
  50. Sebastian Raschka, Learning scikit-learn! 51! http://scikit-learn.org/stable/tutorial/machine_learning_map/! ! Further Resources! Andreas’

    “cheat sheet”!
  51. Sebastian Raschka, Learning scikit-learn! Further Resources! 52! Great “math-free,” practical

    guide to machine learning with scikit-learn! ! ! ! By Andreas Mueller (scikit-learn core developer)! and Sarah Guido! ! http://shop.oreilly.com/product/0636920030515.do! Estimated release: October 20, 2016!
  52. Sebastian Raschka, Learning scikit-learn! 53! Further Resources! My favorite machine

    learning “math & theory” books! h"p://statweb.stanford.edu/~Mbs/ ElemStatLearn/ (free PDF) h"p://www.wiley.com/WileyCDA/WileyTitle/ productCd-0471056693.html
  53. Sebastian Raschka, Learning scikit-learn! My own book.! Some math, !

    “from-scratch” code, ! and practical scikit-learn examples! 54! Further Resources! https://github.com/rasbt/python- machine-learning-book! h"ps://www.amazon.com/Python-Machine- Learning-SebasMan-Raschka/dp/1783555130/ GitHub repository:! Amazon link:!
  54. Sebastian Raschka, Learning scikit-learn! Thanks for attending!! Contact:! o  E-mail:

    mail@sebastianraschka.com! o  Website: http://sebastianraschka.com! o  Twitter: @rasbt! o  GitHub: rasbt! Link to the material:! https://github.com/rasbt/pydata-chicago2016-ml-tutorial! 55!