Share
Embed
Dive deep into machine learning with TensorFlow @ GDG Lisboa Devfest 2017
2017 version with updated features since r0.12, more examples, added codelabs.
https://devfest.gdglisbon.xyz/schedule/day1
Dive deep into machine learning with TensorFlow is a high-level introduction to the TensorFlow library, architecture and API. Originally developed at Google for the purposes of conducting machine-learning and deep neural networks research, TensorFlow leverages a general computational model that is applicable in a wide variety of other domains, especially for performing large-scale numerical computations on large data. This talk explains you how to use the library to train machine-learning models and make your next application smarter.
Speaker:
Mário Cordeiro
Universidade do Porto, Porto, Portugal
Mário
Mário Cordeiro
November 11, 2017
Transcript
81 slides
-
Dive deep into Machine Learning with . Mário Cordeiro GDG
DevFest Lisboa 2017 2017-11-11 [email protected] @mmfcordeiro https://www.pexels.com/photo/underwater-blue-ocean-sea-9149/ -
In this presentation you will learn Introductory concepts of Machine
Learning Few basic concepts of Deep Learning What is TensorFlow ▪ Key differences from other approaches ▪ the dataflow-like model ▪ APIs and High level APIs ▪ TensorBoard features -
In this presentation you will see TensorFlow in action Ready
to use Pre-trained models Transfer Learning techniques What happened in 2 years of TensorFlow Tutorials, Code labs and Videos -
Machine Learning ~= Programming + Data y = f(x) What
is Machine Learning? input prediction Properties: • Generic enough to deal with different domain problems • No specific domain rules encoded model -
Example model for a Linear Regression y = W x
+ b What is Machine Learning? “Learn” means a data-driven procedure to find useful values for parameters input prediction function modeled by 2 parameters -
Example model for a Linear Regression y = W x
+ b What is Machine Learning? input prediction slope -
What is Machine Learning? What are we trying to do?
Mystery equation: y = 0,1x + 0,3 + noise Model: y = Wx + b Objective: given enough (x, y) samples, figure out the value of W and b (x=1,y=3) (x=7,y=9) (x=2,y=5) (x=12,y=20) (x=9,y=11) (…) y=f(x) Data Model -
What is Machine Learning? Let’s code this up Inference: f
(x) { “given x, this is the code that predicts y” } Loss: loss (y, expected_y) {“quantify how bad our prediction was”} Optimize: update (W, b) {“update parameters W, b to improve the prediction”} -
What is Machine Learning? Pet Classifier y = W x
+ b input prediction parameters -
What is Machine Learning? Finding the best features w c
r w c r h h y = W x + b input prediction parameters x y are vectors, folks -
What is Machine Learning? Finding the best features w h
c r w h c r def get_texture_feature ( pixels ): ...? I suspect this won’t work -
What is Machine Learning Deep Learning? No feature engineering needed
Your lazy humans… just use my raw pixels! -
What is Machine Learning Deep Learning? No feature engineering needed
and Neural Networks -
What is Deep Learning? Artificial Neural Network vs Deep Neural
Network pixels Feat Val size: 10 w: 7 h: 7 r: 2,5 c: features First Layers Learn the best features -
What is Deep Learning? Playing the imitation game pixels +
“dog” label Model predicted the Pet correctly = from a labeled Pet dataset shuffle and feed the DNN with Pets -
What is Deep Learning? Playing the imitation game pixels +
“parrot” label Model predicted the Pet incorrectly ≠ back propagate error and continue with training phase weights from a labeled Pet dataset shuffle and feed the DNN with Pets -
What is Deep Learning? Playing the imitation game ▪ Loss
function ▪ Optimization via Gradient Descend pixels + “parrot” label Model predicted the Pet incorrectly ≠ back propagate error and continue with training phase weights -
What is Deep Learning? Playing the imitation game ▪ Loss
function ▪ Optimization via Gradient Descend pixels + “cat” label Model predicted the Pet correctly = weights -
What is Deep Learning? Playing the imitation game ▪ Loss
function ▪ Optimization via Gradient Descend Parameter #1 Parameter #2 Loss -
Why Deep Learning? Artificial Neural Networks (ANN) again? ▪ 50’s
and 60’s ▪ Today’s Quality of Data + Quantity of Data + Computation Power ▪ Better training + faster training ▪ Allow us to build complex models + + Penn Treebank WebTreebank -
Why Deep Learning? Artificial Neural Networks (ANN) again? ▪ 50’s
and 60’s ▪ Today’s Quality of Data + Quantity of Data + Computation Power ▪ Better training + faster training ▪ Allow us to build complex models (Inception-v3) “trained from scratch on a desktop with 8 NVIDIA Tesla K40s in about 2 weeks” https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html 150,000 photographs 1000 object categories -
Why Deep Learning? Current state-of-the-art in ▪ Image: classification, captioning
▪ Language: translation, parsing, summarization ▪ Speech: recognition, generation ▪ Games: AlphaGo, Atari https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html -
Growing use of Deep Learning at Google Web Driven Mobile
Driven AI First -
What is TensorFlow? TensorFlow is an interface for expressing machine
learning algorithms, and an implementation for executing such algorithms. -
What is TensorFlow? A computation expressed using TensorFlow can be
executed with little or no change on a wide variety of heterogeneous systems -
What is TensorFlow? Google Brain project ▪ DistBelief first-generation ▪
scalable distributed training and inference system ▪ TensorFlow is the second generation ▪ Apache 2.0 license Dataflow graph-based computation infrastructure ▪ General purpose (Great for Deep Learning) ▪ Distributed ▪ Parallel Heterogeneous devices ▪ Data Centers, CPUs, GPUs, TPUs and mobile devices -
The dataflow-like model A graph is like a blueprint: ▪
defined using high-level language ▪ compiled and optimized ▪ executed (in parts or fully) on available devices ▪ nodes represent computations ▪ data (tensors) flows between them Tensor: ▪ a scalar is a tensor ▪ a vector is a tensor ▪ a matrix is a tensor ▪ a multidimensional array is a tensor -
Ops: Kernels: TensorFlow architecture TensorFlow Distributed Execution System Add Mul
Print Reshape … CPU GPU TPU Android iOS … C++ front-end ... Compound Ops Go Python front-end Layers Estimator Keras Canned Estimators High Level APIs • Models in a box • Train and evaluate models • Build models -
import tensorflow as tf b = tf.Variable(tf.zeros([100])) # 100-d vector,
init to zeroes W = tf.Variable(tf.random_uniform([784,100],-1,1)) # 784x100 matrix w/rnd vals x = tf.placeholder(name="x") # Placeholder for input relu = tf.nn.relu(tf.matmul(W, x) + b) # Relu(Wx+b) C = [...] # Cost computed as a function # of Relu Code Fragment vs Computation Graph Computation Graph -
# define a training step y_ = tf.placeholder(tf.float32, [None, 10])
xent = -tf.reduce_sum(y_*tf.log(y)) C = tf.train.GradientDescentOptimizer(0.01).minimize(xent) Code Fragment vs Training Step Computation Graph Built-in automatic gradient computation Parameter W Parameter b Loss -
init = tf.initialize_all_variables() # initialize session s = tf.Session() s.run
(init) for step in xrange(0, 10): input = ...construct 100-D input array ... # Create 100-d vector for input result = s.run(C, feed_dict={x: input}) # Fetch cost, feeding x=input print step, result Code Fragment vs Training Iterations Computation Graph Single machine or distributed system structure -
Programming with TensorFlow Python API: Building graphs mostly like numpy
▪ With special operation for deep learning ▪ TF Learn like scikit-learn General Purpose Operations: ▪ Basics: constant, random, placeholder, cast, shape ▪ Variables: assign, assign_sub, assign_add ▪ Queues: enqueue, enqueue_batch, dequeue, blocking or not ▪ Logical: equal, greater, less, where, min, max, argmin, argmax ▪ Tensor computations: all math ops, matmul, determinant, inverse, cholesky ▪ Images: encode, decode, crop, pad, resize, color spaces, random perturbations ▪ Sparse tensors: represented as 3 tensors. ▪ File IO: file reader (sstable, recordio, txt, csv), parser protocol buffers ▪ Control flow: control dependencies, conditionals, loops, functions -
Programming with TensorFlow Neural Net Specific Operations ▪ Activations: sigmoid,
tanh, relu, relu6, elu, dropout ▪ Pooling: avg, max ▪ Convolutions: 1D, 2D, 3D with many options ▪ Normalization: local, batch, moving averages ▪ Classification: softmax, softmax loss, cross entropy loss, topk ▪ Embeddings: distributed lookups/gather, scatter/updates ▪ Sampling: candidate sampler (various options), sampling softmax ▪ Updates: "fused ops" to speed-up optimizer updates (Adagrad, Momentum.) ▪ Summaries: Capture information for visualization -
Programming with TensorFlow High level APIs since r1.0 ▪ tf.layers:
conv2d, conv3d, max_pooling1d, etc ▪ tf.estimator: LinearRegressor, LinearClassifier, DNNRegressor ▪ tf.keras: models, estimators, optimizers, etc ▪ Canned Estimators: cnn, LTSM, etc # Instantiate a Keras inception v3 model. keras_inception_v3 = tf.keras.applications.inception_v3.InceptionV3(weights=None) # Compile model with the optimizer, loss, and metrics you'd like to train with. keras_inception_v3.compile(optimizer=tf.keras.optimizers.SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metric='accuracy') # Create an Estimator from the compiled Keras model. est_inception_v3 = tf.keras.estimator.model_to_estimator(keras_model=keras_inception_v3) # Treat the derived Estimator as you would any other Estimator. For example, # the following derived Estimator calls the train method: est_inception_v3.train(input_fn=my_training_set, steps=2000) -
Programming with TensorFlow High level APIs since r1.0 ▪ tf.layers:
conv2d, conv3d, max_pooling1d, etc ▪ tf.estimator: LinearRegressor, LinearClassifier, DNNRegressor ▪ tf.keras: models, estimators, optimizers, etc ▪ Canned Estimators: cnn, LTSM, etc Best practices and default settings included -
Android MNIST Detect handwritten digits in Android with Tensorflow https://github.com/miyosuda/TensorFlowAndroidMNIST
Accuracy: ▪ Linear ~90% ▪ Two-layer DNN ~96% ▪ ConvNet ~99% (close to the state of the art on this toy problem) -
From Pixels to Actions Human-level control through Deep Reinforcement Learning
https://research.googleblog.com/2015/02/from-pixels-to-actions-human-level.html https://github.com/devsisters/DQN-tensorflow -
From Pixels to Actions “Deep Reinforcement Learning: Playing a Racing
Game” by Pedro Lopes https://lopespm.github.io/machine_learning/2016/10/06/deep-reinforcement-learning-racing-game.html -
AlphaGo Mastering the ancient game of Go with Machine Learning
https://research.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html -
lip-read Google’s DeepMind AI can lip-read TV shows better than
a pro https://www.newscientist.com/article/2113299-googles-deepmind-ai-can-lip-read-tv-shows-better-than-a-pro/ “Our implementation is based on the TensorFlow library and trained on a GeForce Titan X GPU with 12GB memory. […] The model was trained for around 500,000 iterations, which took approximately 10 days.” arXiv, DOI: arXiv:1611.05358v1 -
Universal language translation Google's AI can translate language pairs it
has never seen https://www.engadget.com/2016/11/24/google-ai-translate-language-pairs-it-has-never-seen/ “All training is carried out as in our general NMT pipeline as described in [24] and implemented in TensorFlow.” arXiv, DOI: arXiv:1611.04558v1 -
Deep Koalarization Image colorization using deep CNN and features extracted
from Inception-ResNet-v2 https://github.com/baldassarreFe/deep-koalarization -
EnhanceNet EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis https://arxiv.org/pdf/1612.07919.pdf
input image highest PSNR ENet-PAT -
Create your own artwork Deep Dream, Style Transfer and Inceptionism
https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html https://deepdreamgenerator.com/ https://www.instagram.com/deepdreamgenerator/ http://ostagram.ru/static_pages/lenta?last_days=30 -
Inception Pre-trained model for Image Classification https://research.googleblog.com/2016/08/improving-inception-and-image.html An Alaskan Malamute
(left) and a Siberian Husky (right). Images from Wikipedia -
Show and Tell Pre-trained model for Image Captioning https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html Model
generates a completely new caption using concepts learned from similar scenes in the training set Automatically captioned by the system -
Text summarization with TensorFlow Pre-trained model for Extractive and Abstractive
summarization https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html Original Text: Alice and Bob took the train to visit the zoo. They saw a baby giraffe, a lion, and a flock of colorful tropical birds. Extractive Summary: Alice and Bob visit the zoo. saw a flock of birds. Abstractive summary: Alice and Bob visited the zoo and saw animals and birds -
Transfer Learning Extending pre-trained models for your classes https://codelabs.developers.google.com/codelabs/tensorflow-for-poets https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/08_Transfer_Learning.ipynb
Training Inception v3 takes 2 weeks in 8 NVIDIA Tesla K40s https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html Building the Transfer Values and training just the Fully- Connected Layer takes seconds in regular PC without GPU. -
r0.5 r0.6 r0.7 r0.8 r0.9 r0.10 r0.11 r0.12 r1.0 r1.1
r1.2 r1.3 r1.4 2 Years of TensorFlow Initial Release TensorFlow Serving Python 3.3 Faster on GPUs CuDNN v5 CUDA 8 HDFS XLA Java TF Debugger 58x speedup Windows Go API TensorBoard Embeddings Distributed (multi-machine) based on gRPC iOS support Raspberry Pi (IoT) TF-Slim (tf.contrib.slim) Java API for Windows Keras 2 API to contrib Python 3.6 on Windows Canned estimators • DNNClassifier • DNNRegressor • LinearClassifier • LinearRegressor cuDNN v6 tf.keras and tf.data part of core TensorFlow API Nov ‘15 Nov ‘16 Nov ‘17 Feb ‘17 -
2 Years of TensorFlow Community: ▪ Most popular Machine Learning
project on GitHub http://leotam.github.io/general/2017/04/10/Frameworks.html -
Ready to dive into TensorFlow? Heterogeneous devices Same API ▪
for Researchers, Data Scientists and Developers ▪ from Design to Production Use/Adapt state-of-the-art models Great ecosystem, excellent community support, open source Easy model deployment to devices and Cloud ML Deeplearning4j vs PyTorch vs TensorFlow vs Caffe vs Keras vs MxNet vs Gluon & CNTK https://deeplearning4j.org/compare-dl4j-tensorflow-pytorch -
Ready to dive into TensorFlow? TensorFlow white papers http://download.tensorflow.org/paper/whitepaper2015.pdf https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf
TensorFlow Tutorials https://www.tensorflow.org/tutorials Machine Learning Recipes with Josh Gordon [beginner] https://www.youtube.com/watch?v=Gj0iyo265bc&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal Magnus Erik Hvass Pedersen TensorFlow Tutorials [intermediate] https://github.com/Hvass-Labs/TensorFlow-Tutorials https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ TensorFlow Dev Summit 2017 Keynote and Sessions https://events.withgoogle.com/tensorflow-dev-summit/videos-and-agenda/#content -
TensorFlow for Poets Codelab You will learn: ▪ Transfer Learning
in images You will need: ▪ your laptop (linux, macOS, windows) ▪ Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2/ or 16 Minutes 45 Minutes -
TensorFlow and deep learning, without a PhD You will learn:
▪ recognize handwritten digits (MNIST) You will need: ▪ your laptop (linux, macOS, windows) ▪ Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist or 149 Minutes