Sklearn-Theano -- 2nd Paris Deep Learning Meetup February 18th, 2015

A short presentation of sklearn-theano and how it makes it easy to bring neural net feature extractors into your scikit-learn pipeline.

Transcript

1. Sklearn-Theano Easy and Powerful Feature Extraction for your Scikit-Learn pipeline Michael Eickenberg Equipe Parietal Inria Saclay / Neurospin, CEA Saclay
2. Sklearn-Theano Easy and Powerful Feature Extraction for your Scikit-Learn pipeline Michael Eickenberg Equipe Parietal Inria Saclay / Neurospin, CEA Saclay
3. A simple method for the detection and localization of Bradypus variegatus Michael Eickenberg Equipe Parietal Inria Saclay / Neurospin, CEA Saclay * No animals were harmed making this talk ** Slides HEAVILY borrowed from Kyle Kastner's talk at EuroSciPy2014 -- Thanks!
4. The Plan Classify Locate ??? Business model
5. The Plan Classify Locate ??? Business model This Talk Homework
6. Introductory slides we have known and loved
7. Quick recap: CV before ~2012 Fixed Feature Extractors Simple Classifier SURF SIFT SVM skimage sklearn HOG linear models Feature Aggregation e.g. sklearn Vector Quantization Sparse Coding Spatial Pooling
8. Trainable Feature Extractors Trainable Classifier Neural Net! Theano caffe Trainable Midlevel Features pylearn2 “Deep” Computer vision Quick recap: CV after ~2012 Neural Net!! Neural Net!!! torch torch blocks torch Lasagne ... Not pictured: - Monster GPUs - CPU clusters - 1.2TB of images
9. Convolutional Nets [r8] conv pool conv pool [1],[2],[3]
10. Sklearn-theano
11. Sklearn-theano
12. Sklearn - largest ML library - fast and efficient - simple, flat design - intuitive core API - python - open source - pip install sklearn Theano - Optimizing compiler - Math expressions - python - writes like numpy - Automatic diff! - several backends - CPU, GPU, switch with one flag!
13. Sklearn from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline pipeline = make_pipeline(StandardScaler(), PCA(whiten=True, n_components=10), LogisticRegression()) pipeline.fit(X_train, y_train) y_pred = pipeline.predict(X_test)
14. Sklearn from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline pipeline = make_pipeline(StandardScaler(), PCA(whiten=True, n_components=10), LogisticRegression()) pipeline.fit(X_train, y_train) y_pred = pipeline.predict(X_test)
15. from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline pipeline = make_pipeline(StandardScaler(), PCA(whiten=True, n_components=10), LogisticRegression()) pipeline.fit(X_train, y_train) y_pred = pipeline.predict(X_test) Sklearn Get powerful pretrained nets into the pipeline as feature extractors
16. Theano import theano import theano.tensor as T import numpy.random.randn as randn shapes = [(1000, 728), (1000, 1), (1000, 1000), (1000, 1), (10, 1000), (10, 1)] W1, b1, W2, b2, W3, b3 = map(theano.shared, [randn(*shape) for shape in shapes]) def relu(expr): return T.maximum(expr, 0) inp = T.fmatrix() net = T.nnet.softmax(W3.dot(relu(W2.dot(relu(W1.dot(inp) + b1)) + b2)) + b3) f = theano.function([inp], net)
17. sklearn-theano • Kyle Kastner (U Montreal) and myself • Easy interface to trained networks • Bring deep nets into sklearn pipelines • Includes examples from this talk https://github.com/sklearn-theano/sklearn-theano
18. sklearn-theano Trained Feature Extractor Trainable Classifier Canny sklearn Other pretrained models OverFeat (NYU)[4] DeCAF (UCB) Canny Not pictured: Weeks of expert time and compute time Cross-validation of entire networks! linear models SVM sklearn- theano
19. sklearn-theano features OverFeat
20. sklearn-theano Trained Feature Extractor Trainable Classifier Ca nn y from sklearn.pipeline import make_pipeline from sklearn.linear_model import \ LogisticRegression from sklearn_theano.feature_extraction import ( OverfeatTransformer, OverfeatClassifier) pipeline1 = OverfeatClassifier() pipeline2 = make_pipeline(OverfeatTransformer(output_layer=[-3]), LogisticRegression()) for ImageNet concepts only: General purpose: Use features to learn a new task
21. Classification sklearn_theano/examples/plot_classification.py
22. Localization sklearn_theano/examples/plot_localization_tutorial.py
23. sklearn-theano features GAN
24. Generative Adversarial Nets • Ian Goodfellow and many others, 2014 [5] • Trains two networks ◦ Generate examples from random input ◦ Discriminate whether generated or real Generation sklearn_theano/examples/plot_mnist_generator.py
25. sklearn-theano features ...
26. sklearn-theano features GoogLeNet!
27. sklearn-theano features GoogLeNet
28. BVLC Caffe GoogLeNet - There is also pycaffe, please check it out! - Why translate Caffe models to Theano? - Make it really easy to use out of the box with sklearn! - Bring Caffe into the Theano world! - immediate access to all layers of state of the art pre- trained nets in your theano structures! - For many other things, it is really worth installing and using Caffe!
29. Takeaways • Scikit-Learn is an amazing ML package (but no deep nets) • Theano is a great package for {C,G}PU math in Python • Neural networks are powerful feature extractors • Use large neural nets easily in sklearn-theano ◦ Use them within sklearn pipelines ◦ Use them as building blocks for your own Theano net
30. Future Work • More loaders for state of the art convolutional nets • Support other types of inputs, e.g. audio and text • Enable fine tuning to specific tasks • Help us bring powerful feature extractors into the scikit- learn and Theano worlds! • Scattering transform
31. Code: https://github.com/sklearn-theano/sklearn-theano Thank You! Friend us on TweetHub @kastnerkyle @meickenberg @kastnerkyle @eickenberg
32. References [1] http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial [2] http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf [3] http://parse.ele.tue.nl/education/cluster2 [4] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, Y. LeCun. OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks, International Conference on Learning Representations (ICLR 2014), April 2014. http://arxiv.org/abs/1406.2661 [5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative Adversarial Networks, arXiV, June 2014. http://arxiv.org/abs/1406.2661 [6] Going Deeper With Convolutions, C. Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, September 2014. http://arxiv.org/abs/1409.4842
33. BONUS SLIDES
34. Preprocessing class precision recall f1-score support 0.0 0.89 0.89 0.89 9 1.0 0.91 0.91 0.91 11 avg / total 0.90 0.90 0.90 20 accuracy score 0.9 tf = OverfeatTransformer(output_layers=[-3]) clf = LogisticRegression() pipe = make_pipeline(tf, clf) pipe.fit(X_train, y_train) y_pred = pipe.predict(X_test) Code sklearn-theano/examples/asirra_dataset_example.py