Dive deep into machine learning with TensorFlow @ GDG Lisboa Devfest 2017

2017 version with updated features since r0.12, more examples, added codelabs.

https://devfest.gdglisbon.xyz/schedule/day1

Dive deep into machine learning with TensorFlow is a high-level introduction to the TensorFlow library, architecture and API. Originally developed at Google for the purposes of conducting machine-learning and deep neural networks research, TensorFlow leverages a general computational model that is applicable in a wide variety of other domains, especially for performing large-scale numerical computations on large data. This talk explains you how to use the library to train machine-learning models and make your next application smarter.

Speaker:

Mário Cordeiro
Universidade do Porto, Porto, Portugal
Mário

Transcript

1. Dive deep into Machine Learning with . Mário Cordeiro GDG DevFest Lisboa 2017 2017-11-11 mario.miguel.cordeiro@gmail.com @mmfcordeiro https://www.pexels.com/photo/underwater-blue-ocean-sea-9149/
2. In this presentation you will learn Introductory concepts of Machine Learning Few basic concepts of Deep Learning What is TensorFlow ▪ Key differences from other approaches ▪ the dataflow-like model ▪ APIs and High level APIs ▪ TensorBoard features
3. In this presentation you will see TensorFlow in action Ready to use Pre-trained models Transfer Learning techniques What happened in 2 years of TensorFlow Tutorials, Code labs and Videos
4. http://insights.dice.com/2016/09/26/deep-learning-core-technology/ What is
5. Machine Learning ~= Programming + Data y = f(x) What is Machine Learning? input prediction Properties: • Generic enough to deal with different domain problems • No specific domain rules encoded model
6. Example model for a Linear Regression y = W x + b What is Machine Learning? “Learn” means a data-driven procedure to find useful values for parameters input prediction function modeled by 2 parameters
7. Example model for a Linear Regression y = W x + b What is Machine Learning? input prediction slope
8. What is Machine Learning? What are we trying to do? Mystery equation: y = 0,1x + 0,3 + noise Model: y = Wx + b Objective: given enough (x, y) samples, figure out the value of W and b (x=1,y=3) (x=7,y=9) (x=2,y=5) (x=12,y=20) (x=9,y=11) (…) y=f(x) Data Model
9. What is Machine Learning? Let’s code this up Inference: f (x) { “given x, this is the code that predicts y” } Loss: loss (y, expected_y) {“quantify how bad our prediction was”} Optimize: update (W, b) {“update parameters W, b to improve the prediction”}
10. What is Machine Learning? Pet Classifier y = W x + b input prediction parameters
11. What is Machine Learning? Finding the best features w c r w c r h h y = W x + b input prediction parameters x y are vectors, folks
12. What is Machine Learning? Finding the best features w h c r w h c r def get_texture_feature ( pixels ): ...? I suspect this won’t work
13. What is Machine Learning? Problem: Many variations in same class!
14. What is Machine Learning? Problem: Many similarities between classes!
15. What is Machine Learning? Textures are Pixels def get_texture_feature ( pixels ): ...?
16. What is Machine Learning Deep Learning? No feature engineering needed Your lazy humans… just use my raw pixels!
17. What is Machine Learning Deep Learning? No feature engineering needed and Neural Networks
18. What is Deep Learning? Artificial Neural Network vs Deep Neural Network
19. What is Deep Learning? Artificial Neural Network vs Deep Neural Network pixels Feat Val size: 10 w: 7 h: 7 r: 2,5 c: features First Layers Learn the best features
20. What is Deep Learning? Playing the imitation game pixels + “dog” label Model predicted the Pet correctly = from a labeled Pet dataset shuffle and feed the DNN with Pets
21. What is Deep Learning? Playing the imitation game pixels + “parrot” label Model predicted the Pet incorrectly ≠ back propagate error and continue with training phase weights from a labeled Pet dataset shuffle and feed the DNN with Pets
22. What is Deep Learning? Playing the imitation game ▪ Loss function ▪ Optimization via Gradient Descend pixels + “parrot” label Model predicted the Pet incorrectly ≠ back propagate error and continue with training phase weights
23. What is Deep Learning? Playing the imitation game ▪ Loss function ▪ Optimization via Gradient Descend pixels + “cat” label Model predicted the Pet correctly = weights
24. What is Deep Learning? Playing the imitation game ▪ Loss function ▪ Optimization via Gradient Descend Parameter #1 Parameter #2 Loss
25. What is Deep Learning? http://www.businessinsider.com/what-is-google-tensorflow-2015-11
26. TensorFlowPlayground http://playground.tensorflow.org
27. Why Deep Learning? Artificial Neural Networks (ANN) again? ▪ 50’s and 60’s ▪ Today’s Quality of Data + Quantity of Data + Computation Power ▪ Better training + faster training ▪ Allow us to build complex models + + Penn Treebank WebTreebank
28. Why Deep Learning? Artificial Neural Networks (ANN) again? ▪ 50’s and 60’s ▪ Today’s Quality of Data + Quantity of Data + Computation Power ▪ Better training + faster training ▪ Allow us to build complex models (Inception-v3) “trained from scratch on a desktop with 8 NVIDIA Tesla K40s in about 2 weeks” https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html 150,000 photographs 1000 object categories
29. Why Deep Learning? Current state-of-the-art in ▪ Image: classification, captioning ▪ Language: translation, parsing, summarization ▪ Speech: recognition, generation ▪ Games: AlphaGo, Atari https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html
30. https://events.linuxfoundation.org/sites/events/files/slides/Machine_Intelligence_at_Google_Scale-_Vision-Speech_API__TensorFlow_and_Cloud_ML.pdf
31. Growing use of Deep Learning at Google
32. Growing use of Deep Learning at Google
33. Growing use of Deep Learning at Google
34. Growing use of Deep Learning at Google
35. Growing use of Deep Learning at Google Web Driven Mobile Driven AI First
36. What is TensorFlow?
37. What is TensorFlow? TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms.
38. What is TensorFlow? A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems
39. What is TensorFlow? A graph of operations. A multidimensional array.
40. What is TensorFlow? Google Brain project ▪ DistBelief first-generation ▪ scalable distributed training and inference system ▪ TensorFlow is the second generation ▪ Apache 2.0 license Dataflow graph-based computation infrastructure ▪ General purpose (Great for Deep Learning) ▪ Distributed ▪ Parallel Heterogeneous devices ▪ Data Centers, CPUs, GPUs, TPUs and mobile devices
41. The dataflow-like model A graph is like a blueprint: ▪ defined using high-level language ▪ compiled and optimized ▪ executed (in parts or fully) on available devices ▪ nodes represent computations ▪ data (tensors) flows between them Tensor: ▪ a scalar is a tensor ▪ a vector is a tensor ▪ a matrix is a tensor ▪ a multidimensional array is a tensor
42. Ops: Kernels: TensorFlow architecture TensorFlow Distributed Execution System Add Mul Print Reshape … CPU GPU TPU Android iOS … C++ front-end ... Compound Ops Go Python front-end Layers Estimator Keras Canned Estimators High Level APIs • Models in a box • Train and evaluate models • Build models
43. Under the hood https://www.wired.com/2015/11/google-open-sourcing-tensorflow-shows-ais-future-is-data-not-code/
44. import tensorflow as tf b = tf.Variable(tf.zeros([100])) # 100-d vector, init to zeroes W = tf.Variable(tf.random_uniform([784,100],-1,1)) # 784x100 matrix w/rnd vals x = tf.placeholder(name="x") # Placeholder for input relu = tf.nn.relu(tf.matmul(W, x) + b) # Relu(Wx+b) C = [...] # Cost computed as a function # of Relu Code Fragment vs Computation Graph Computation Graph
45. # define a training step y_ = tf.placeholder(tf.float32, [None, 10]) xent = -tf.reduce_sum(y_*tf.log(y)) C = tf.train.GradientDescentOptimizer(0.01).minimize(xent) Code Fragment vs Training Step Computation Graph Built-in automatic gradient computation Parameter W Parameter b Loss
46. init = tf.initialize_all_variables() # initialize session s = tf.Session() s.run (init) for step in xrange(0, 10): input = ...construct 100-D input array ... # Create 100-d vector for input result = s.run(C, feed_dict={x: input}) # Fetch cost, feeding x=input print step, result Code Fragment vs Training Iterations Computation Graph Single machine or distributed system structure
47. Machine Learning using TensorFlow http://terencebroad.com/nnvis.html
48. Programming with TensorFlow Python API: Building graphs mostly like numpy ▪ With special operation for deep learning ▪ TF Learn like scikit-learn General Purpose Operations: ▪ Basics: constant, random, placeholder, cast, shape ▪ Variables: assign, assign_sub, assign_add ▪ Queues: enqueue, enqueue_batch, dequeue, blocking or not ▪ Logical: equal, greater, less, where, min, max, argmin, argmax ▪ Tensor computations: all math ops, matmul, determinant, inverse, cholesky ▪ Images: encode, decode, crop, pad, resize, color spaces, random perturbations ▪ Sparse tensors: represented as 3 tensors. ▪ File IO: file reader (sstable, recordio, txt, csv), parser protocol buffers ▪ Control flow: control dependencies, conditionals, loops, functions
49. Programming with TensorFlow Neural Net Specific Operations ▪ Activations: sigmoid, tanh, relu, relu6, elu, dropout ▪ Pooling: avg, max ▪ Convolutions: 1D, 2D, 3D with many options ▪ Normalization: local, batch, moving averages ▪ Classification: softmax, softmax loss, cross entropy loss, topk ▪ Embeddings: distributed lookups/gather, scatter/updates ▪ Sampling: candidate sampler (various options), sampling softmax ▪ Updates: "fused ops" to speed-up optimizer updates (Adagrad, Momentum.) ▪ Summaries: Capture information for visualization
50. Programming with TensorFlow High level APIs since r1.0 ▪ tf.layers: conv2d, conv3d, max_pooling1d, etc ▪ tf.estimator: LinearRegressor, LinearClassifier, DNNRegressor ▪ tf.keras: models, estimators, optimizers, etc ▪ Canned Estimators: cnn, LTSM, etc # Instantiate a Keras inception v3 model. keras_inception_v3 = tf.keras.applications.inception_v3.InceptionV3(weights=None) # Compile model with the optimizer, loss, and metrics you'd like to train with. keras_inception_v3.compile(optimizer=tf.keras.optimizers.SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metric='accuracy') # Create an Estimator from the compiled Keras model. est_inception_v3 = tf.keras.estimator.model_to_estimator(keras_model=keras_inception_v3) # Treat the derived Estimator as you would any other Estimator. For example, # the following derived Estimator calls the train method: est_inception_v3.train(input_fn=my_training_set, steps=2000)
51. Programming with TensorFlow High level APIs since r1.0 ▪ tf.layers: conv2d, conv3d, max_pooling1d, etc ▪ tf.estimator: LinearRegressor, LinearClassifier, DNNRegressor ▪ tf.keras: models, estimators, optimizers, etc ▪ Canned Estimators: cnn, LTSM, etc Best practices and default settings included
52. TensorBoard Interactive Computation Graph Visualization
53. TensorBoard Visualizing Learning http://localhost:6006 tensorboard --logdir /tmp/retrain_logs
54. TensorBoard Embedding Visualization for high dimensional data
55. TensorFlow in action
56. Android MNIST Detect handwritten digits in Android with Tensorflow https://github.com/miyosuda/TensorFlowAndroidMNIST Accuracy: ▪ Linear ~90% ▪ Two-layer DNN ~96% ▪ ConvNet ~99% (close to the state of the art on this toy problem)
57. From Pixels to Actions Human-level control through Deep Reinforcement Learning https://research.googleblog.com/2015/02/from-pixels-to-actions-human-level.html https://github.com/devsisters/DQN-tensorflow
58. From Pixels to Actions “Deep Reinforcement Learning: Playing a Racing Game” by Pedro Lopes https://lopespm.github.io/machine_learning/2016/10/06/deep-reinforcement-learning-racing-game.html
59. AlphaGo Mastering the ancient game of Go with Machine Learning https://research.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html
60. lip-read Google’s DeepMind AI can lip-read TV shows better than a pro https://www.newscientist.com/article/2113299-googles-deepmind-ai-can-lip-read-tv-shows-better-than-a-pro/ “Our implementation is based on the TensorFlow library and trained on a GeForce Titan X GPU with 12GB memory. […] The model was trained for around 500,000 iterations, which took approximately 10 days.” arXiv, DOI: arXiv:1611.05358v1
61. Universal language translation Google's AI can translate language pairs it has never seen https://www.engadget.com/2016/11/24/google-ai-translate-language-pairs-it-has-never-seen/ “All training is carried out as in our general NMT pipeline as described in [24] and implemented in TensorFlow.” arXiv, DOI: arXiv:1611.04558v1
62. Deep Koalarization Image colorization using deep CNN and features extracted from Inception-ResNet-v2 https://github.com/baldassarreFe/deep-koalarization
63. EnhanceNet EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis https://arxiv.org/pdf/1612.07919.pdf input image highest PSNR ENet-PAT
64. Create your own artwork Deep Dream, Style Transfer and Inceptionism https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html https://deepdreamgenerator.com/ https://www.instagram.com/deepdreamgenerator/ http://ostagram.ru/static_pages/lenta?last_days=30
65. Pre-trained Models
66. Inception Pre-trained model for Image Classification https://research.googleblog.com/2016/08/improving-inception-and-image.html An Alaskan Malamute (left) and a Siberian Husky (right). Images from Wikipedia
67. Show and Tell Pre-trained model for Image Captioning https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html Model generates a completely new caption using concepts learned from similar scenes in the training set Automatically captioned by the system
68. ParseyMcParseface Pre-trained models for language parsing (NLP part-of-speech tagging) https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html
69. Text summarization with TensorFlow Pre-trained model for Extractive and Abstractive summarization https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html Original Text: Alice and Bob took the train to visit the zoo. They saw a baby giraffe, a lion, and a flock of colorful tropical birds. Extractive Summary: Alice and Bob visit the zoo. saw a flock of birds. Abstractive summary: Alice and Bob visited the zoo and saw animals and birds
70. Transfer Learning Extending pre-trained models for your classes https://codelabs.developers.google.com/codelabs/tensorflow-for-poets https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/08_Transfer_Learning.ipynb Training Inception v3 takes 2 weeks in 8 NVIDIA Tesla K40s https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html Building the Transfer Values and training just the Fully- Connected Layer takes seconds in regular PC without GPU.
71. 2 Years of TensorFlow
72. r0.5 r0.6 r0.7 r0.8 r0.9 r0.10 r0.11 r0.12 r1.0 r1.1 r1.2 r1.3 r1.4 2 Years of TensorFlow Initial Release TensorFlow Serving Python 3.3 Faster on GPUs CuDNN v5 CUDA 8 HDFS XLA Java TF Debugger 58x speedup Windows Go API TensorBoard Embeddings Distributed (multi-machine) based on gRPC iOS support Raspberry Pi (IoT) TF-Slim (tf.contrib.slim) Java API for Windows Keras 2 API to contrib Python 3.6 on Windows Canned estimators • DNNClassifier • DNNRegressor • LinearClassifier • LinearRegressor cuDNN v6 tf.keras and tf.data part of core TensorFlow API Nov ‘15 Nov ‘16 Nov ‘17 Feb ‘17
73. 2 Years of TensorFlow Community: ▪ Most popular Machine Learning project on GitHub http://leotam.github.io/general/2017/04/10/Frameworks.html
74. 2 Years of TensorFlow https://twitter.com/fchollet/status/915626952408436736
75. https://www.pexels.com/photo/black-and-white-lake-dive-header-4465/ Ready to dive into TensorFlow?
76. Ready to dive into TensorFlow? Heterogeneous devices Same API ▪ for Researchers, Data Scientists and Developers ▪ from Design to Production Use/Adapt state-of-the-art models Great ecosystem, excellent community support, open source Easy model deployment to devices and Cloud ML Deeplearning4j vs PyTorch vs TensorFlow vs Caffe vs Keras vs MxNet vs Gluon & CNTK https://deeplearning4j.org/compare-dl4j-tensorflow-pytorch
77. Ready to dive into TensorFlow? TensorFlow white papers http://download.tensorflow.org/paper/whitepaper2015.pdf https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf TensorFlow Tutorials https://www.tensorflow.org/tutorials Machine Learning Recipes with Josh Gordon [beginner] https://www.youtube.com/watch?v=Gj0iyo265bc&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal Magnus Erik Hvass Pedersen TensorFlow Tutorials [intermediate] https://github.com/Hvass-Labs/TensorFlow-Tutorials https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ TensorFlow Dev Summit 2017 Keynote and Sessions https://events.withgoogle.com/tensorflow-dev-summit/videos-and-agenda/#content
78. TensorFlow for Poets Codelab You will learn: ▪ Transfer Learning in images You will need: ▪ your laptop (linux, macOS, windows) ▪ Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2/ or 16 Minutes 45 Minutes
79. TensorFlow and deep learning, without a PhD You will learn: ▪ recognize handwritten digits (MNIST) You will need: ▪ your laptop (linux, macOS, windows) ▪ Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist or 149 Minutes
80. Data Science Community in Portugal https://www.meetup.com/datascienceportugal https://www.facebook.com/datascienceportugal https://datascienceportugal.slack.com
81. Thank you @mmfcordeiro mario.miguel.cordeiro@gmail.com https://www.pexels.com/photo/black-and-white-lake-dive-header-4465/