Share
Embed
Sklearn-Theano -- 2nd Paris Deep Learning Meetup February 18th, 2015
A short presentation of sklearn-theano and how it makes it easy to bring neural net feature extractors into your scikit-learn pipeline.
eickenberg
February 18, 2015
Transcript
34 slides
-
Sklearn-Theano Easy and Powerful Feature Extraction for your Scikit-Learn pipeline
Michael Eickenberg Equipe Parietal Inria Saclay / Neurospin, CEA Saclay -
Sklearn-Theano Easy and Powerful Feature Extraction for your Scikit-Learn pipeline
Michael Eickenberg Equipe Parietal Inria Saclay / Neurospin, CEA Saclay -
A simple method for the detection and localization of Bradypus
variegatus Michael Eickenberg Equipe Parietal Inria Saclay / Neurospin, CEA Saclay * No animals were harmed making this talk ** Slides HEAVILY borrowed from Kyle Kastner's talk at EuroSciPy2014 -- Thanks! -
Quick recap: CV before ~2012 Fixed Feature Extractors Simple Classifier
SURF SIFT SVM skimage sklearn HOG linear models Feature Aggregation e.g. sklearn Vector Quantization Sparse Coding Spatial Pooling -
Trainable Feature Extractors Trainable Classifier Neural Net! Theano caffe Trainable
Midlevel Features pylearn2 “Deep” Computer vision Quick recap: CV after ~2012 Neural Net!! Neural Net!!! torch torch blocks torch Lasagne ... Not pictured: - Monster GPUs - CPU clusters - 1.2TB of images -
Sklearn - largest ML library - fast and efficient -
simple, flat design - intuitive core API - python - open source - pip install sklearn Theano - Optimizing compiler - Math expressions - python - writes like numpy - Automatic diff! - several backends - CPU, GPU, switch with one flag! -
Sklearn from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from
sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline pipeline = make_pipeline(StandardScaler(), PCA(whiten=True, n_components=10), LogisticRegression()) pipeline.fit(X_train, y_train) y_pred = pipeline.predict(X_test) -
Sklearn from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from
sklearn.linear_model import LogisticRegression from sklearn.pipeline import make_pipeline pipeline = make_pipeline(StandardScaler(), PCA(whiten=True, n_components=10), LogisticRegression()) pipeline.fit(X_train, y_train) y_pred = pipeline.predict(X_test) -
from sklearn.preprocessing import StandardScaler from sklearn.decomposition import PCA from sklearn.linear_model
import LogisticRegression from sklearn.pipeline import make_pipeline pipeline = make_pipeline(StandardScaler(), PCA(whiten=True, n_components=10), LogisticRegression()) pipeline.fit(X_train, y_train) y_pred = pipeline.predict(X_test) Sklearn Get powerful pretrained nets into the pipeline as feature extractors -
Theano import theano import theano.tensor as T import numpy.random.randn as
randn shapes = [(1000, 728), (1000, 1), (1000, 1000), (1000, 1), (10, 1000), (10, 1)] W1, b1, W2, b2, W3, b3 = map(theano.shared, [randn(*shape) for shape in shapes]) def relu(expr): return T.maximum(expr, 0) inp = T.fmatrix() net = T.nnet.softmax(W3.dot(relu(W2.dot(relu(W1.dot(inp) + b1)) + b2)) + b3) f = theano.function([inp], net) -
sklearn-theano • Kyle Kastner (U Montreal) and myself • Easy
interface to trained networks • Bring deep nets into sklearn pipelines • Includes examples from this talk https://github.com/sklearn-theano/sklearn-theano -
sklearn-theano Trained Feature Extractor Trainable Classifier Canny sklearn Other pretrained
models OverFeat (NYU)[4] DeCAF (UCB) Canny Not pictured: Weeks of expert time and compute time Cross-validation of entire networks! linear models SVM sklearn- theano -
sklearn-theano Trained Feature Extractor Trainable Classifier Ca nn y from
sklearn.pipeline import make_pipeline from sklearn.linear_model import \ LogisticRegression from sklearn_theano.feature_extraction import ( OverfeatTransformer, OverfeatClassifier) pipeline1 = OverfeatClassifier() pipeline2 = make_pipeline(OverfeatTransformer(output_layer=[-3]), LogisticRegression()) for ImageNet concepts only: General purpose: Use features to learn a new task -
Generative Adversarial Nets • Ian Goodfellow and many others, 2014
[5] • Trains two networks ◦ Generate examples from random input ◦ Discriminate whether generated or real Generation sklearn_theano/examples/plot_mnist_generator.py -
BVLC Caffe GoogLeNet - There is also pycaffe, please check
it out! - Why translate Caffe models to Theano? - Make it really easy to use out of the box with sklearn! - Bring Caffe into the Theano world! - immediate access to all layers of state of the art pre- trained nets in your theano structures! - For many other things, it is really worth installing and using Caffe! -
Takeaways • Scikit-Learn is an amazing ML package (but no
deep nets) • Theano is a great package for {C,G}PU math in Python • Neural networks are powerful feature extractors • Use large neural nets easily in sklearn-theano ◦ Use them within sklearn pipelines ◦ Use them as building blocks for your own Theano net -
Future Work • More loaders for state of the art
convolutional nets • Support other types of inputs, e.g. audio and text • Enable fine tuning to specific tasks • Help us bring powerful feature extractors into the scikit- learn and Theano worlds! • Scattering transform -
References [1] http://deeplearning.stanford.edu/wiki/index.php/UFLDL_Tutorial [2] http://www.cs.nyu.edu/~yann/talks/lecun-ranzato-icml2013.pdf [3] http://parse.ele.tue.nl/education/cluster2 [4] P. Sermanet,
D. Eigen, X. Zhang, M. Mathieu, R. Fergus, Y. LeCun. OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks, International Conference on Learning Representations (ICLR 2014), April 2014. http://arxiv.org/abs/1406.2661 [5] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio. Generative Adversarial Networks, arXiV, June 2014. http://arxiv.org/abs/1406.2661 [6] Going Deeper With Convolutions, C. Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, Andrew Rabinovich, September 2014. http://arxiv.org/abs/1409.4842 -
Preprocessing class precision recall f1-score support 0.0 0.89 0.89 0.89
9 1.0 0.91 0.91 0.91 11 avg / total 0.90 0.90 0.90 20 accuracy score 0.9 tf = OverfeatTransformer(output_layers=[-3]) clf = LogisticRegression() pipe = make_pipeline(tf, clf) pipe.fit(X_train, y_train) y_pred = pipe.predict(X_test) Code sklearn-theano/examples/asirra_dataset_example.py