Machine Learning With Scikit-Learn ODSC SF 2015

Introduction to machine learning with scikit-learn. Material at https://github.com/amueller/odscon-sf-2015

Transcript

1. Machine Learning with Scikit-Learn
   Andreas Mueller (NYU Center for Data Science, scikit-learn)
   Material: http://bit.ly/sklsf
   

   View Slide

2. 2
   Me
   

   View Slide

3. 3
   Classification
   Regression
   Clustering
   Semi-Supervised Learning
   Feature Selection
   Feature Extraction
   Manifold Learning
   Dimensionality Reduction
   Kernel Approximation
   Hyperparameter Optimization
   Evaluation Metrics
   Out-of-core learning
   …...
   

   View Slide

4. View Slide

5. 5
   Get the notebooks!
   http://bit.ly/sklsf
   

   View Slide

6. 6
   http://scikit-learn.org/
   

   View Slide

7. 7
   Hi Andy,
   I just received an email from the first tutorial
   speaker, presenting right before you, saying
   he's ill and won't be able to make it.
   I know you have already committed yourself to
   two presentations, but is there anyway you
   could increase your tutorial time slot, maybe
   just offer time to try out what you've taught?
   Otherwise I have to do some kind of modern
   dance interpretation of Python in data :-)
   -Leah
   Hi Andreas,
   I am very interested in your Machine Learning
   background. I work for X Recruiting who have
   been engaged by Z, a worldwide leading supplier
   of Y. We are expanding the core engineering
   team and we are looking for really passionate
   engineers who want to create their own story and
   help millions of people.
   Can we find a time for a call to chat for a few
   minutes about this?
   Thanks
   

   View Slide

8. 8
   Hi Andy,
   I just received an email from the first tutorial
   speaker, presenting right before you, saying
   he's ill and won't be able to make it.
   I know you have already committed yourself to
   two presentations, but is there anyway you
   could increase your tutorial time slot, maybe
   just offer time to try out what you've taught?
   Otherwise I have to do some kind of modern
   dance interpretation of Python in data :-)
   -Leah
   Hi Andreas,
   I am very interested in your Machine Learning
   background. I work for X Recruiting who have
   been engaged by Z, a worldwide leading supplier
   of Y. We are expanding the core engineering
   team and we are looking for really passionate
   engineers who want to create their own story and
   help millions of people.
   Can we find a time for a call to chat for a few
   minutes about this?
   Thanks
   

   View Slide

9. 9
   Doing Machine Learning With Scikit-Learn
   

   View Slide

10. 10
    Representing Data
    X =
    1.1 2.2 3.4 5.6 1.0
    6.7 0.5 0.4 2.6 1.6
    2.4 9.3 7.3 6.4 2.8
    1.5 0.0 4.3 8.3 3.4
    0.5 3.5 8.1 3.6 4.6
    5.1 9.7 3.5 7.9 5.1
    3.7 7.8 2.6 3.2 6.3
    

    View Slide

11. 11
    Representing Data
    X =
    1.1 2.2 3.4 5.6 1.0
    6.7 0.5 0.4 2.6 1.6
    2.4 9.3 7.3 6.4 2.8
    1.5 0.0 4.3 8.3 3.4
    0.5 3.5 8.1 3.6 4.6
    5.1 9.7 3.5 7.9 5.1
    3.7 7.8 2.6 3.2 6.3
    one sample
    

    View Slide

12. 12
    Representing Data
    X =
    1.1 2.2 3.4 5.6 1.0
    6.7 0.5 0.4 2.6 1.6
    2.4 9.3 7.3 6.4 2.8
    1.5 0.0 4.3 8.3 3.4
    0.5 3.5 8.1 3.6 4.6
    5.1 9.7 3.5 7.9 5.1
    3.7 7.8 2.6 3.2 6.3
    one sample
    one feature
    

    View Slide

13. 13
    Representing Data
    X = y =
    1.1 2.2 3.4 5.6 1.0
    6.7 0.5 0.4 2.6 1.6
    2.4 9.3 7.3 6.4 2.8
    1.5 0.0 4.3 8.3 3.4
    0.5 3.5 8.1 3.6 4.6
    5.1 9.7 3.5 7.9 5.1
    3.7 7.8 2.6 3.2 6.3
    1.6
    2.7
    4.4
    0.5
    0.2
    5.6
    6.7
    one sample
    one feature outputs / labels
    

    View Slide

14. 14
    Training and Testing Data
    X =
    1.1 2.2 3.4 5.6 1.0
    6.7 0.5 0.4 2.6 1.6
    2.4 9.3 7.3 6.4 2.8
    1.5 0.0 4.3 8.3 3.4
    0.5 3.5 8.1 3.6 4.6
    5.1 9.7 3.5 7.9 5.1
    3.7 7.8 2.6 3.2 6.3
    y =
    1.6
    2.7
    4.4
    0.5
    0.2
    5.6
    6.7
    

    View Slide

15. 15
    Training and Testing Data
    X =
    1.1 2.2 3.4 5.6 1.0
    6.7 0.5 0.4 2.6 1.6
    2.4 9.3 7.3 6.4 2.8
    1.5 0.0 4.3 8.3 3.4
    0.5 3.5 8.1 3.6 4.6
    5.1 9.7 3.5 7.9 5.1
    3.7 7.8 2.6 3.2 6.3
    y =
    1.6
    2.7
    4.4
    0.5
    0.2
    5.6
    6.7
    training set
    test set
    

    View Slide

16. 16
    Training and Testing Data
    X =
    1.1 2.2 3.4 5.6 1.0
    6.7 0.5 0.4 2.6 1.6
    2.4 9.3 7.3 6.4 2.8
    1.5 0.0 4.3 8.3 3.4
    0.5 3.5 8.1 3.6 4.6
    5.1 9.7 3.5 7.9 5.1
    3.7 7.8 2.6 3.2 6.3
    y =
    1.6
    2.7
    4.4
    0.5
    0.2
    5.6
    6.7
    training set
    test set
    from sklearn.cross_validation import train_test_split
    X_train, X_test, y_train, y_test = train_test_split(X, y)
    

    View Slide

17. 17
    Supervised Machine Learning
    Training Data
    Training Labels
    Model
    

    View Slide

18. 18
    Supervised Machine Learning
    Training Data
    Test Data
    Training Labels
    Model
    Prediction
    

    View Slide

19. 19
    Supervised Machine Learning
    Training Data
    Test Data
    Training Labels
    Model
    Prediction
    Test Labels Evaluation
    

    View Slide

20. 20
    Supervised Machine Learning
    Training Data
    Test Data
    Training Labels
    Model
    Prediction
    Test Labels Evaluation
    Training
    Generalization
    

    View Slide

21. 21
    clf = RandomForestClassifier()
    clf.fit(X_train, y_train)
    Training Data
    Training Labels
    Model
    

    View Slide

22. 22
    clf = RandomForestClassifier()
    clf.fit(X_train, y_train)
    Training Data
    Test Data
    Training Labels
    Model
    Prediction
    y_pred = clf.predict(X_test)
    

    View Slide

23. 23
    clf = RandomForestClassifier()
    clf.fit(X_train, y_train)
    clf.score(X_test, y_test)
    Training Data
    Test Data
    Training Labels
    Model
    Prediction
    Test Labels Evaluation
    y_pred = clf.predict(X_test)
    

    View Slide

24. 24
    IPython Notebook:
    Part 1 - Introduction to Scikit-learn
    

    View Slide

25. 25
    Unsupervised Machine Learning
    Training Data Model
    

    View Slide

26. 26
    Unsupervised Machine Learning
    Training Data
    Test Data
    Model
    New View
    

    View Slide

27. 27
    pca = PCA()
    pca.fit(X_train)
    X_new = pca.transform(X_test)
    Training Data
    Test Data
    Model
    Transformation
    Unsupervised Transformations
    

    View Slide

28. 28
    IPython Notebook:
    Part 2 – Unsupervised Transformers
    

    View Slide

29. 29
    Basic API
    estimator.fit(X, [y])
    estimator.predict estimator.transform
    Classification Preprocessing
    Regression Dimensionality reduction
    Clustering Feature selection
    Feature extraction
    

    View Slide

30. 30
    All Data
    Training data Test data
    

    View Slide

31. 31
    All Data
    Training data Test data
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    

    View Slide

32. 32
    All Data
    Training data Test data
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Split 1
    

    View Slide

33. 33
    All Data
    Training data Test data
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Split 1
    Split 2
    

    View Slide

34. 34
    All Data
    Training data Test data
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Split 1
    Split 2
    Split 3
    Split 4
    Split 5
    

    View Slide

35. 35
    IPython Notebook:
    Part 3 - Cross-validation
    

    View Slide

36. View Slide

37. View Slide

38. 38
    All Data
    Training data Test data
    

    View Slide

39. 39
    All Data
    Training data Test data
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Test data
    Split 1
    Split 2
    Split 3
    Split 4
    Split 5
    

    View Slide

40. 40
    All Data
    Training data Test data
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
    Test data
    Finding Parameters
    Final evaluation
    Split 1
    Split 2
    Split 3
    Split 4
    Split 5
    

    View Slide

41. 41
    SVC(C=0.001,
    gamma=0.001)
    

    View Slide

42. 42
    SVC(C=0.001,
    gamma=0.001)
    SVC(C=0.01,
    gamma=0.001)
    SVC(C=0.1,
    gamma=0.001)
    SVC(C=1,
    gamma=0.001)
    SVC(C=10,
    gamma=0.001)
    

    View Slide

43. 43
    SVC(C=0.001,
    gamma=0.001)
    SVC(C=0.01,
    gamma=0.001)
    SVC(C=0.1,
    gamma=0.001)
    SVC(C=1,
    gamma=0.001)
    SVC(C=10,
    gamma=0.001)
    SVC(C=0.001,
    gamma=0.01)
    SVC(C=0.01,
    gamma=0.01)
    SVC(C=0.1,
    gamma=0.01)
    SVC(C=1,
    gamma=0.01)
    SVC(C=10,
    gamma=0.01)
    

    View Slide

44. 44
    SVC(C=0.001,
    gamma=0.001)
    SVC(C=0.01,
    gamma=0.001)
    SVC(C=0.1,
    gamma=0.001)
    SVC(C=1,
    gamma=0.001)
    SVC(C=10,
    gamma=0.001)
    SVC(C=0.001,
    gamma=0.01)
    SVC(C=0.01,
    gamma=0.01)
    SVC(C=0.1,
    gamma=0.01)
    SVC(C=1,
    gamma=0.01)
    SVC(C=10,
    gamma=0.01)
    SVC(C=0.001,
    gamma=0.1)
    SVC(C=0.01,
    gamma=0.1)
    SVC(C=0.1,
    gamma=0.1)
    SVC(C=1,
    gamma=0.1)
    SVC(C=10,
    gamma=0.1)
    

    View Slide

45. 45
    SVC(C=0.001,
    gamma=0.001)
    SVC(C=0.01,
    gamma=0.001)
    SVC(C=0.1,
    gamma=0.001)
    SVC(C=1,
    gamma=0.001)
    SVC(C=10,
    gamma=0.001)
    SVC(C=0.001,
    gamma=0.01)
    SVC(C=0.01,
    gamma=0.01)
    SVC(C=0.1,
    gamma=0.01)
    SVC(C=1,
    gamma=0.01)
    SVC(C=10,
    gamma=0.01)
    SVC(C=0.001,
    gamma=0.1)
    SVC(C=0.01,
    gamma=0.1)
    SVC(C=0.1,
    gamma=0.1)
    SVC(C=1,
    gamma=0.1)
    SVC(C=10,
    gamma=0.1)
    SVC(C=0.001,
    gamma=1)
    SVC(C=0.01,
    gamma=1)
    SVC(C=0.1,
    gamma=1)
    SVC(C=1,
    gamma=1)
    SVC(C=10,
    gamma=1)
    SVC(C=0.001,
    gamma=10)
    SVC(C=0.01,
    gamma=10)
    SVC(C=0.1,
    gamma=10)
    SVC(C=1,
    gamma=10)
    SVC(C=10,
    gamma=10)
    

    View Slide

46. 46
    IPython Notebook:
    Part 4 – Grid Searches
    

    View Slide

47. 47
    Training Data
    Training Labels
    Model
    

    View Slide

48. 48
    Training Data
    Training Labels
    Model
    

    View Slide

49. 49
    Training Data
    Training Labels
    Model
    Feature
    Extraction
    

    View Slide

50. 50
    Training Data
    Training Labels
    Model
    Feature
    Extraction
    Scaling
    

    View Slide

51. 51
    Training Data
    Training Labels
    Model
    Feature
    Extraction
    Scaling
    Feature
    Selection
    

    View Slide

52. 52
    Training Data
    Training Labels
    Model
    Feature
    Extraction
    Scaling
    Feature
    Selection
    Cross Validation
    

    View Slide

53. 53
    Training Data
    Training Labels
    Model
    Feature
    Extraction
    Scaling
    Feature
    Selection
    Cross Validation
    

    View Slide

54. 54
    Pipelines
    

    View Slide

55. 55
    Pipelines
    

    View Slide

56. 56
    IPython Notebook:
    Part 5 - Preprocessing and Pipelines
    

    View Slide

57. 57
    Do cross-validation over all steps jointly.
    Keep a separate test set until the very end.
    

    View Slide

58. 58
    Sample application: Sentiment Analysis
    

    View Slide

59. 59
    Review:
    One of the worst movies I've ever rented. Sorry it had one of my
    favorite actors on it (Travolta) in a nonsense role. In fact, anything
    made sense in this movie.
    Who can say there was true love between Eddy and Maureen?
    Don't you remember the beginning of the movie ?
    Is she so lovely? Ask her daughters. I don't think so.
    Label: negative
    Training data: 12500 positive, 12500 negative
    IMDB Movie Reviews Data
    

    View Slide

60. 60
    Bag Of Word Representations
    CountVectorizer / TfidfVectorizer
    

    View Slide

61. 61
    Bag Of Word Representations
    “This is how you get ants.”
    CountVectorizer / TfidfVectorizer
    

    View Slide

62. 62
    Bag Of Word Representations
    “This is how you get ants.”
    ['this', 'is', 'how', 'you', 'get', 'ants']
    CountVectorizer / TfidfVectorizer
    tokenizer
    

    View Slide

63. 63
    Bag Of Word Representations
    “This is how you get ants.”
    ['this', 'is', 'how', 'you', 'get', 'ants']
    CountVectorizer / TfidfVectorizer
    tokenizer
    Build a vocabulary over all documents
    ['aardvak', 'amsterdam', 'ants', ... 'you', 'your', 'zyxst']
    

    View Slide

64. 64
    Bag Of Word Representations
    “This is how you get ants.”
    [0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ]
    ants get you
    aardvak zyxst
    ['this', 'is', 'how', 'you', 'get', 'ants']
    CountVectorizer / TfidfVectorizer
    tokenizer
    Sparse matrix encoding
    Build a vocabulary over all documents
    ['aardvak', 'amsterdam', 'ants', ... 'you', 'your', 'zyxst']
    

    View Slide

65. 65
    IPython Notebook:
    Part 6 - Working With Text Data
    

    View Slide

66. 66
    Feature Union
    Training Data
    Training Labels
    Model
    Feature
    Extraction I
    Feature
    Extraction II
    

    View Slide

67. 67
    IPython Notebook:
    Part 7 – FeatureUnion
    

    View Slide

68. 68
    Overfitting and Underfitting
    Model complexity
    Accuracy
    Training
    

    View Slide

69. 69
    Overfitting and Underfitting
    Model complexity
    Accuracy
    Training
    Generalization
    

    View Slide

70. 70
    Overfitting and Underfitting
    Model complexity
    Accuracy
    Training
    Generalization
    Underfitting Overfitting
    Sweet spot
    

    View Slide

71. 71
    Linear SVM
    

    View Slide

72. 72
    Linear SVM
    

    View Slide

73. 73
    (RBF) Kernel SVM
    

    View Slide

74. 74
    (RBF) Kernel SVM
    

    View Slide

75. 75
    (RBF) Kernel SVM
    

    View Slide

76. 76
    (RBF) Kernel SVM
    

    View Slide

77. 77
    Decision Trees
    

    View Slide

78. 78
    Decision Trees
    

    View Slide

79. 79
    Decision Trees
    

    View Slide

80. 80
    Decision Trees
    

    View Slide

81. 81
    Decision Trees
    

    View Slide

82. 82
    Decision Trees
    

    View Slide

83. 83
    Random Forests
    

    View Slide

84. 84
    Random Forests
    

    View Slide

85. 85
    Random Forests
    

    View Slide

86. 86
    Validation Curves
    train_scores, test_scores = validation_curve(SVC(), X, y,
    param_name="gamma", param_range=param_range)
    

    View Slide

87. Andreas Mueller 87
    Scaling Up
    

    View Slide

88. Andreas Mueller 88
    Three regimes of data
    ●
    Fits in RAM
    ●
    Fits on a Hard Drive
    ●
    Doesn't fit on a single PC
    

    View Slide

89. Andreas Mueller 89
    Three regimes of data
    ●
    Fits in RAM (up to 256 GB?)
    ●
    Fits on a Hard Drive (up to 6TB?)
    ●
    Doesn't fit on a single PC
    

    View Slide

90. Andreas Mueller 90
    

    View Slide

91. Andreas Mueller 91
    

    View Slide

92. Andreas Mueller 92
    "256Gb ought to be enough for anybody."
    - me
    

    View Slide

93. Andreas Mueller 93
    "256Gb ought to be enough for anybody."
    - me
    (for machine learning)
    

    View Slide

94. Andreas Mueller 94
    Subsample!
    

    View Slide

95. Andreas Mueller 95
    The scikit-learn way
    

    View Slide

96. Andreas Mueller 96
    HDD
    Network
    estimator.partial_fit(X_batch, y_batch)
    Your for-loop / polling
    Trained
    Scikit-learn
    estimator
    

    View Slide

97. 97
    Supported Algorithms
    ●
    All SGDClassifier derivatives
    ●
    Naive Bayes
    ●
    MinibatchKMeans
    ●
    Birch
    ●
    IncrementalPCA
    ●
    MiniBatchDictionaryLearning
    

    View Slide

98. 98
    IPython Notebook:
    Part 8 – Out Of Core Learning
    

    View Slide

99. 99
    Stateless Transformers
    ●
    Normalizer
    ●
    HashingVectorizer
    ●
    RBFSampler (and other kernel approx)
    

    View Slide

100. 100
     Bag Of Word Representations
     “This is how you get ants.”
     [0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ]
     ants get you
     aardvak zyxst
     ['this', 'is', 'how', 'you', 'get', 'ants']
     CountVectorizer / TfidfVectorizer
     tokenizer
     Sparse matrix encoding
     Build a vocabulary over all documents
     ['aardvak', 'amsterdam', 'ants', ... 'you',
     'your', 'zyxst']
     

     View Slide

101. 101
     Hashing Trick
     “This is how you get ants.”
     [0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ]
     ants get you
     aardvak zyxst
     ['this', 'is', 'how', 'you', 'get', 'ants']
     HashingVectorizer
     tokenizer
     Sparse matrix encoding
     hashing
     [hash('this'), hash('is'), hash('how'), hash('you'),
     hash('get'), hash('ants')]
     = [832412, 223788, 366226, 81185, 835749, 173092]
     

     View Slide

102. 102
     IPython Notebook:
     Part 9 – Out Of Core Learning for Text
     

     View Slide

103. 103
     Video Series
     Advanced Machine Learning with scikit-learn
     50% Off Coupon Code: AUTHD
     

     View Slide

104. 104
     Video Series
     Advanced Machine Learning with scikit-learn
     50% Off Coupon Code: AUTHD
     

     View Slide

105. 105
     Thank you for your attention.
     @t3kcit
     @amueller
     importamueller@gmail.com
     http://amueller.github.io
     

     View Slide