Share
Embed
Dive deep into machine learning with TensorFlow @ GDG Porto Devfest 2016
Dive deep into machine learning with TensorFlow is a high-level introduction to the TensorFlow library, architecture and API. Originally developed at Google for the purposes of conducting machine-learning and deep neural networks research, TensorFlow leverages a general computational model that is applicable in a wide variety of other domains, especially for performing large-scale numerical computations on large data. This talk explains you how to use the library to train machine-learning models and make your next application smarter.
Speakers
Mário Cordeiro
Universidade do Porto, Porto, Portugal
Mário
Mário Cordeiro
December 03, 2016
Transcript
73 slides
-
Dive deep into machine learning with . Mário Cordeiro Porto
GDG DevFest 2016 2016-12-02 [email protected] @mmfcordeiro https://www.pexels.com/photo/underwater-blue-ocean-sea-9149/ -
In this presentation you will learn Basic of ML and
Deep Learning What is TensorFlow, the dataflow-like model, API and TensorBoard TensorFlow in action, please! Pre-trained models and Learning Transfer To evaluate the right deep learning library for you -
Machine Learning ~= Programming + Data y = W x
+ b What is Machine Learning? “Learn” means a data-driven procedure to find useful values for parameters input prediction parameters -
Linear Regression y = W x + b What is
Machine Learning? input prediction slope -
What is Machine Learning? What are we trying to do?
Mystery equation: y = 0,1x + 0,3 + noise Model: y = Wx + b Objective: given enough (x, y) samples, figure out the value of W and b -
What is Machine Learning? Let’s code this up Inference: f
(x) { “given x, this is the code that predicts y” } Loss: loss (y, expected_y) {“quantify how bad our prediction was”} Optimize: update (W, b) {“update parameters W, b to improve the prediction”} -
What is Machine Learning? Finding the best features w h
c r w h c r def get_texture_feature ( pixels ): ...? -
What is Machine Learning Deep Learning? No feature engineering needed
What!? Neural Networks just use my raw pixels -
What is Deep Learning? Playing the imitation game pixels Feat
Val size: 10 w: 7 h: 7 r: 2,5 c: features -
What is Deep Learning? Playing the imitation game pixels +
“orange” label Model predicted the fruit correctly = from a labeled fruit dataset shuffle and feed the DNN with fruits -
What is Deep Learning? Playing the imitation game pixels +
“pineapple” label Model predicted the fruit incorrectly ≠ Back propagate error and continue weights from a labeled fruit dataset shuffle and feed the DNN with fruits -
What is Deep Learning? Playing the imitation game Loss
function Optimization via Gradient Descend pixels + “pineapple” label Model predicted the fruit incorrectly ≠ Back propagate error and continue weights -
What is Deep Learning? Playing the imitation game Loss
function Optimization via Gradient Descend pixels + “apple” label Model predicted the fruit correctly = weights -
What is Deep Learning? Playing the imitation game Loss
function Optimization via Gradient Descend Parameter #1 Parameter #2 Loss -
Why Deep Learning? Artificial Neural Networks (ANN) again? 50’s
and 60’s Today’s Quality of Data + Quantity of Data + Computation Power Better training + faster training Allow us to build complex models + + Penn Treebank WebTreebank -
Why Deep Learning? Artificial Neural Networks (ANN) again? 50’s
and 60’s Today’s Quality of Data + Quantity of Data + Computation Power Better training + faster training Allow us to build complex models “trained from scratch on a desktop with 8 NVIDIA Tesla K40s in about 2 weeks” https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html 150,000 photographs 1000 object categories -
Why Deep Learning? Current state-of-the-art in Image: classification, captioning
Language: translation, parsing, summarization Speech: recognition, generation Games: AlphaGo, Atari -
What is TensorFlow? TensorFlow is an interface for expressing machine
learning algorithms, and an implementation for executing such algorithms. -
What is TensorFlow? A computation expressed using TensorFlow can be
executed with little or no change on a wide variety of heterogeneous systems -
What is TensorFlow? Google Brain project DistBelief first-generation
scalable distributed training and inference system TensorFlow is the second generation Apache 2.0 license Dataflow graph-based computation infrastructure General purpose (Great for Deep Learning) Distributed Parallel Heterogeneous devices Data Centers, CPUs, GPUs, TPUs and mobile devices -
The dataflow-like model A graph is like a blueprint:
defined using high-level language compiled and optimized executed (in parts or fully) on available devices nodes represent computations data (tensors) flows between them Tensor: a scalar is a tensor a vector is a tensor a matrix is a tensor a multidimensional array is a tensor -
Ops: Kernels: TensorFlow architecture TensorFlow Distributed Execution System TensorFlow Distributed
Execution System Add Add Mul Mul Print Print Reshape Reshape … … CPU CPU GPU GPU TPU TPU Android Android iOS iOS … … C++ front-end C++ front-end Python front-end Python front-end ... Compound Ops -
import tensorflow as tf b = tf.Variable(tf.zeros([100])) # 100-d vector,
init to zeroes W = tf.Variable(tf.random_uniform([784,100],-1,1)) # 784x100 matrix w/rnd vals x = tf.placeholder(name="x") # Placeholder for input relu = tf.nn.relu(tf.matmul(W, x) + b) # Relu(Wx+b) C = [...] # Cost computed as a function # of Relu Code Fragment vs Computation Graph Computation Graph -
# define a training step y_ = tf.placeholder(tf.float32, [None, 10])
xent = -tf.reduce_sum(y_*tf.log(y)) C = tf.train.GradientDescentOptimizer(0.01).minimize(xent) Code Fragment vs Training Step Computation Graph Built-in automatic gradient computation Parameter W Parameter b Loss -
init = tf.initialize_all_variables() # initialize session s = tf.Session() s.run
(init) for step in xrange(0, 10): input = ...construct 100-D input array ... # Create 100-d vector for input result = s.run(C, feed_dict={x: input}) # Fetch cost, feeding x=input print step, result Code Fragment vs Training Iterations Computation Graph Single machine or distributed system structure -
Programming with TensorFlow Python API: Building graphs mostly like numpy
With special operation for deep learning TF Learn like scikit-learn General Purpose Operations: Basics: constant, random, placeholder, cast, shape Variables: assign, assign_sub, assign_add Queues: enqueue, enqueue_batch, dequeue, blocking or not Logical: equal, greater, less, where, min, max, argmin, argmax Tensor computations: all math ops, matmul, determinant, inverse, cholesky Images: encode, decode, crop, pad, resize, color spaces, random perturbations Sparse tensors: represented as 3 tensors. File IO: file reader (sstable, recordio, txt, csv), parser protocol buffers Control flow: control dependencies, conditionals, loops, functions -
Programming with TensorFlow Neural Net Operations Activations: sigmoid, tanh,
relu, relu6, elu, dropout Pooling: avg, max Convolutions: 1D, 2D, 3D with many options Normalization: local, batch, moving averages Classification: softmax, softmax loss, cross entropy loss, topk Embeddings: distributed lookups/gather, scatter/updates Sampling: candidate sampler (various options), sampling softmax Updates: "fused ops" to speed-up optimizer updates (Adagrad, Momentum.) Summaries: Capture information for visualization -
Android MNIST Detect handwritten digits in Android with Tensorflow https://github.com/miyosuda/TensorFlowAndroidMNIST
Accuracy: Linear ~90% Two-layer DNN ~96% ConvNet ~99% (close to the state of the art on this toy problem) -
From Pixels to Actions Human-level control through Deep Reinforcement Learning
https://research.googleblog.com/2015/02/from-pixels-to-actions-human-level.html https://github.com/devsisters/DQN-tensorflow -
AlphaGo Mastering the ancient game of Go with Machine Learning
https://research.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html -
lip-read Google’s DeepMind AI can lip-read TV shows better than
a pro https://www.newscientist.com/article/2113299-googles-deepmind-ai-can-lip-read-tv-shows-better-than-a-pro/ “Our implementation is based on the TensorFlow library and trained on a GeForce Titan X GPU with 12GB memory. […] The model was trained for around 500,000 iterations, which took approximately 10 days.” arXiv, DOI: arXiv:1611.05358v1 -
Universal language translation Google's AI can translate language pairs it
has never seen https://www.engadget.com/2016/11/24/google-ai-translate-language-pairs-it-has-never-seen/ “All training is carried out as in our general NMT pipeline as described in [24] and implemented in TensorFlow.” arXiv, DOI: arXiv:1611.04558v1 -
Create your own artwork Deep Dream, Style Transfer and Inceptionism
https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html https://deepdreamgenerator.com/ https://www.instagram.com/deepdreamgenerator/ http://ostagram.ru/static_pages/lenta?last_days=30 -
Inception Pre-trained model for Image Classification https://research.googleblog.com/2016/08/improving-inception-and-image.html An Alaskan Malamute
(left) and a Siberian Husky (right). Images from Wikipedia -
Show and Tell Pre-trained model for Image Captioning https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html Model
generates a completely new caption using concepts learned from similar scenes in the training set Automatically captioned by the system -
Text summarization with TensorFlow Pre-trained model for Extractive and Abstractive
summarization https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html Original Text: Alice and Bob took the train to visit the zoo. They saw a baby giraffe, a lion, and a flock of colorful tropical birds. Extractive Summary: Alice and Bob visit the zoo. saw a flock of birds. Abstractive summary: Alice and Bob visited the zoo and saw animals and birds -
A Year of TensorFlow Releases: r0.6: Python 3.3, Faster on
GPUs r0.7: TensorFlow Serving r0.8: Distributed (multi-machine) support based on gRPC r0.9: iOS support, Raspberry Pi (IoT) r0.10: TF-Slim (tf.contrib.slim) r0.11: CuDNN v5, CUDA 8 support, HDFS Support -
A Year of TensorFlow Community: Most popular Machine Learning
project on GitHub -
Ready to dive into TensorFlow? Heterogonous devices Same API for
Researchers, Data Scientists and Developers Reuse state-of-the-art models Apart from being great ecosystem has excellent community support Easy model deployment to devices and Cloud ML Evaluation of Deep Learning Toolkits https://github.com/zer0n/deepframeworks/blob/master/README.md DL4J vs. Torch vs. Theano vs. Caffe vs. TensorFlow https://deeplearning4j.org/compare-dl4j-torch7-pylearn -
Resources TensorFlow white paper http://download.tensorflow.org/paper/whitepaper2015.pdf TensorFlow Tutorials https://www.tensorflow.org/versions/r0.12/tutorials/index.html Machine Learning
Recipes with Josh Gordon [beginner] https://www.youtube.com/watch?v=Gj0iyo265bc&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal Magnus Erik Hvass Pedersen TensorFlow Tutorials [intermediate] https://github.com/Hvass-Labs/TensorFlow-Tutorials -
TensorFlow for Poets Codelab 15:15 @ Workshops Room You will
need: your laptop (linux, macOS, windows) Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ -
Welcome to Magenta! Machine intelligence for music and art generation
https://magenta.tensorflow.org/welcome-to-magenta -
Google Cloud ML Platform Managed Scalable Machine Learning Platform
Prediction at Scale Build Machine Learning Models Easily Fully Managed Platform Power by TensorFlow APIs Image Analysis Speech Recognition Translation