Machine Learning With Scikit-Learn

A basic intro to scikit-learn introduction, held at OSDS Conference 2015.

1. Machine Learning with Scikit-Learn Andreas Mueller (NYU Center for Data Science, scikit-learn) http://bit.ly/sklodsc
2. 2 Me
3. 3 Classification Regression Clustering Semi-Supervised Learning Feature Selection Feature Extraction Manifold Learning Dimensionality Reduction Kernel Approximation Hyperparameter Optimization Evaluation Metrics Out-of-core learning …...
5. 5 Get the notebooks! http://bit.ly/sklosdc
6. 6 http://scikit-learn.org/
7. 7 Hi Andy, I just received an email from the first tutorial speaker, presenting right before you, saying he's ill and won't be able to make it. I know you have already committed yourself to two presentations, but is there anyway you could increase your tutorial time slot, maybe just offer time to try out what you've taught? Otherwise I have to do some kind of modern dance interpretation of Python in data :-) -Leah Hi Andreas, I am very interested in your Machine Learning background. I work for X Recruiting who have been engaged by Z, a worldwide leading supplier of Y. We are expanding the core engineering team and we are looking for really passionate engineers who want to create their own story and help millions of people. Can we find a time for a call to chat for a few minutes about this? Thanks
8. 8 Hi Andy, I just received an email from the first tutorial speaker, presenting right before you, saying he's ill and won't be able to make it. I know you have already committed yourself to two presentations, but is there anyway you could increase your tutorial time slot, maybe just offer time to try out what you've taught? Otherwise I have to do some kind of modern dance interpretation of Python in data :-) -Leah Hi Andreas, I am very interested in your Machine Learning background. I work for X Recruiting who have been engaged by Z, a worldwide leading supplier of Y. We are expanding the core engineering team and we are looking for really passionate engineers who want to create their own story and help millions of people. Can we find a time for a call to chat for a few minutes about this? Thanks
9. 9 Supervised Machine Learning Training Data Training Labels Model
10. 10 Supervised Machine Learning Training Data Test Data Training Labels Model Prediction
11. 11 Supervised Machine Learning Training Data Test Data Training Labels Model Prediction Test Labels Evaluation
12. 12 Supervised Machine Learning Training Data Test Data Training Labels Model Prediction Test Labels Evaluation Training Generalization
13. 13 clf = RandomForestClassifier() clf.fit(X_train, y_train) Training Data Training Labels Model
14. 14 clf = RandomForestClassifier() clf.fit(X_train, y_train) Training Data Test Data Training Labels Model Prediction y_pred = clf.predict(X_test)
15. 15 clf = RandomForestClassifier() clf.fit(X_train, y_train) clf.score(X_test, y_test) Training Data Test Data Training Labels Model Prediction Test Labels Evaluation y_pred = clf.predict(X_test)
16. 16 IPython Notebook: Part 1 - Introduction to Scikit-learn
17. 17 Unsupervised Machine Learning Training Data Model
18. 18 Unsupervised Machine Learning Training Data Test Data Model New View
19. 19 pca = PCA() pca.fit(X_train) X_new = pca.transform(X_test) Training Data Test Data Model Transformation Unsupervised Transformations
20. 20 IPython Notebook: Part 2 – Unsupervised Transformers
21. 21 Basic API estimator.fit(X, [y]) estimator.predict estimator.transform Classification Preprocessing Regression Dimensionality reduction Clustering Feature selection Feature extraction
22. 22 All Data Training data Test data
23. 23 All Data Training data Test data Fold 1 Fold 2 Fold 3 Fold 4 Fold 5
24. 24 All Data Training data Test data Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Split 1
25. 25 All Data Training data Test data Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Split 1 Split 2
26. 26 All Data Training data Test data Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Split 1 Split 2 Split 3 Split 4 Split 5
27. 27 IPython Notebook: Part 3 - Cross-validation
30. 30 All Data Training data Test data
31. 31 All Data Training data Test data Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Test data Split 1 Split 2 Split 3 Split 4 Split 5
32. 32 All Data Training data Test data Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Test data Finding Parameters Final evaluation Split 1 Split 2 Split 3 Split 4 Split 5
33. 33 SVC(C=0.001, gamma=0.001)
34. 34 SVC(C=0.001, gamma=0.001) SVC(C=0.01, gamma=0.001) SVC(C=0.1, gamma=0.001) SVC(C=1, gamma=0.001) SVC(C=10, gamma=0.001)
35. 35 SVC(C=0.001, gamma=0.001) SVC(C=0.01, gamma=0.001) SVC(C=0.1, gamma=0.001) SVC(C=1, gamma=0.001) SVC(C=10, gamma=0.001) SVC(C=0.001, gamma=0.01) SVC(C=0.01, gamma=0.01) SVC(C=0.1, gamma=0.01) SVC(C=1, gamma=0.01) SVC(C=10, gamma=0.01)
36. 36 SVC(C=0.001, gamma=0.001) SVC(C=0.01, gamma=0.001) SVC(C=0.1, gamma=0.001) SVC(C=1, gamma=0.001) SVC(C=10, gamma=0.001) SVC(C=0.001, gamma=0.01) SVC(C=0.01, gamma=0.01) SVC(C=0.1, gamma=0.01) SVC(C=1, gamma=0.01) SVC(C=10, gamma=0.01) SVC(C=0.001, gamma=0.1) SVC(C=0.01, gamma=0.1) SVC(C=0.1, gamma=0.1) SVC(C=1, gamma=0.1) SVC(C=10, gamma=0.1)
37. 37 SVC(C=0.001, gamma=0.001) SVC(C=0.01, gamma=0.001) SVC(C=0.1, gamma=0.001) SVC(C=1, gamma=0.001) SVC(C=10, gamma=0.001) SVC(C=0.001, gamma=0.01) SVC(C=0.01, gamma=0.01) SVC(C=0.1, gamma=0.01) SVC(C=1, gamma=0.01) SVC(C=10, gamma=0.01) SVC(C=0.001, gamma=0.1) SVC(C=0.01, gamma=0.1) SVC(C=0.1, gamma=0.1) SVC(C=1, gamma=0.1) SVC(C=10, gamma=0.1) SVC(C=0.001, gamma=1) SVC(C=0.01, gamma=1) SVC(C=0.1, gamma=1) SVC(C=1, gamma=1) SVC(C=10, gamma=1) SVC(C=0.001, gamma=10) SVC(C=0.01, gamma=10) SVC(C=0.1, gamma=10) SVC(C=1, gamma=10) SVC(C=10, gamma=10)
38. 38 IPython Notebook: Part 4 – Grid Searches
39. 39 Training Data Training Labels Model
40. 40 Training Data Training Labels Model
41. 41 Training Data Training Labels Model Feature Extraction
42. 42 Training Data Training Labels Model Feature Extraction Scaling
43. 43 Training Data Training Labels Model Feature Extraction Scaling Feature Selection
44. 44 Training Data Training Labels Model Feature Extraction Scaling Feature Selection Cross Validation
45. 45 Training Data Training Labels Model Feature Extraction Scaling Feature Selection Cross Validation
46. 46 IPython Notebook: Part 5 - Preprocessing and Pipelines
47. 47 Do cross-validation over all steps jointly. Keep a separate test set until the very end.
48. 48 Bag Of Word Representations CountVectorizer / TfidfVectorizer
49. 49 Bag Of Word Representations “This is how you get ants.” CountVectorizer / TfidfVectorizer
50. 50 Bag Of Word Representations “This is how you get ants.” ['this', 'is', 'how', 'you', 'get', 'ants'] CountVectorizer / TfidfVectorizer tokenizer
51. 51 Bag Of Word Representations “This is how you get ants.” ['this', 'is', 'how', 'you', 'get', 'ants'] CountVectorizer / TfidfVectorizer tokenizer Build a vocabulary over all documents ['aardvak', 'amsterdam', 'ants', ... 'you', 'your', 'zyxst']
52. 52 Bag Of Word Representations “This is how you get ants.” [0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ] ants get you aardvak zyxst ['this', 'is', 'how', 'you', 'get', 'ants'] CountVectorizer / TfidfVectorizer tokenizer Sparse matrix encoding Build a vocabulary over all documents ['aardvak', 'amsterdam', 'ants', ... 'you', 'your', 'zyxst']
53. 53 Application: Insult detection
54. 54 Application: Insult detection i really don't understand your point. It seems that you are mixing apples and oranges.
55. 55 Application: Insult detection Clearly you're a fucktard. i really don't understand your point. It seems that you are mixing apples and oranges.
56. 56 IPython Notebook: Part 6 - Working With Text Data
57. 57 Feature Union Training Data Training Labels Model Feature Extraction I Feature Extraction II
58. 58 IPython Notebook: Part 7 – FeatureUnion
59. 59 Overfitting and Underfitting Model complexity Accuracy Training
60. 60 Overfitting and Underfitting Model complexity Accuracy Training Generalization
61. 61 Overfitting and Underfitting Model complexity Accuracy Training Generalization Underfitting Overfitting Sweet spot
62. 62 Linear SVM
63. 63 Linear SVM
64. 64 (RBF) Kernel SVM
65. 65 (RBF) Kernel SVM
66. 66 (RBF) Kernel SVM
67. 67 (RBF) Kernel SVM
68. 68 Decision Trees
69. 69 Decision Trees
70. 70 Decision Trees
71. 71 Decision Trees
72. 72 Decision Trees
73. 73 Decision Trees
74. 74 Random Forests
75. 75 Random Forests
76. 76 Random Forests
77. 77 Validation Curves train_scores, test_scores = validation_curve(SVC(), X, y, param_name="gamma", param_range=param_range)
78. 78 train_sizes, train_scores, test_scores = learning_curve( estimator, X, y,train_sizes=train_sizes) Learning Curves
79. 79 Randomized Parameter Search
80. 80 Randomized Parameter Search Source: Bergstra and Bengio
81. 81 Randomized Parameter Search Source: Bergstra and Bengio Step-size free for continuous parameters Decouples runtime from search-space size Robust against irrelevant parameters
82. 82 Randomized Parameter Search • Always use distributions for continuous variables. • Don't use for low dimensional spaces. • Future: Bayesian optimization based search.
83. 83 IPython Notebook: Part 8 – Randomized Parameter Search
84. 84 Generalized Cross-Validation and Path Algorithms
85. 85 IPython Notebook: Part 9 – Generalized Cross-Validation
86. 86 Linear Models Feature Selection Tree-Based models [possible] LogisticRegressionCV [new] RFECV [DecisionTreeCV] RidgeCV [RandomForestClassifierCV] RidgeClassifierCV [GradientBoostingClassifierCV] LarsCV ElasticNetCV ...
87. 87 Scoring Functions
88. 88 Default: Accuracy (classification) R2 (regression) GridSeachCV RandomizedSearchCV cross_val_score ...CV
89. 89 IPython Notebook: Part 10 – Scoring Metrics
90. 90 Out of Core Learning
91. 91 Or: save ourself the effort
92. 92 Think twice! • Old laptop: 4GB Ram • 1073741824 float32 • Or 1mio data points with 1000 features • EC2 : 256 GB Ram • 68719476736 float32 • Or 68mio data points with 1000 features
93. 93 Supported Algorithms • All SGDClassifier derivatives • Naive Bayes • MinibatchKMeans • Birch • IncrementalPCA • MiniBatchDictionaryLearning
94. 94 IPython Notebook: Part 11 – Out Of Core Learning
95. 95 Stateless Transformers • Normalizer • HashingVectorizer • RBFSampler (and other kernel approx)
96. 96 Bag Of Word Representations “This is how you get ants.” [0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ] ants get you aardvak zyxst ['this', 'is', 'how', 'you', 'get', 'ants'] CountVectorizer / TfidfVectorizer tokenizer Sparse matrix encoding Build a vocabulary over all documents ['aardvak', 'amsterdam', 'ants', ... 'you', 'your', 'zyxst']
97. 97 Hashing Trick “This is how you get ants.” [0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ] ants get you aardvak zyxst ['this', 'is', 'how', 'you', 'get', 'ants'] HashingVectorizer tokenizer Sparse matrix encoding hashing [hash('this'), hash('is'), hash('how'), hash('you'), hash('get'), hash('ants')] = [832412, 223788, 366226, 81185, 835749, 173092]
98. 98 IPython Notebook: Part 12 – Out Of Core Learning for Text
99. 99 http://scikit-learn.org/
100. 100 CDS is hiring Research Engineers
101. 101 Thank you for your attention. @t3kcit @amueller importamueller@gmail.com http://amueller.github.io