Dive deep into machine learning with TensorFlow @ Checkmarx, Braga, 2018

2018 version with updated new features since r1.4, more examples, added codelabs.

hands-on Examples with Docker Zeppelin Notebook

1. Linear Regression Normal Equation
2. Linear regression using batch gradient descent
3. Classification (Softmax Regression, or Multinomial Logistic Regression)
4. scikit-learn intro
5. Optical Character Recognition (simple logistic regression)
6. MNIST 1.0 (TensorFlow) with 1-layer neural network
7. MNIST 2.0 (TensorFlow) using 5 fully connected layers with a softmax
8. MNIST 3.0 (TensorFlow) using convolutional network
9. MNIST 4.0 (Keras + TensorFlow)
10. Inception Model and Transfer Learning

Run Docker:

docker pull mmfcordeiro/zeppelin-in-docker-with-tensorflow
docker run -p 8080:8080 -p 4040:4040 -p 8888:8888 -p 6006:6006 mmfcordeiro/zeppelin-in-docker-with-tensorflow

Open Zeppelin:

Zeppelin open in http://localhost:8080
Tensorboard open in http://localhost:6006

Dive deep into machine learning with TensorFlow is a high-level introduction to the TensorFlow library, architecture and API. Originally developed at Google for the purposes of conducting machine-learning and deep neural networks research, TensorFlow leverages a general computational model that is applicable in a wide variety of other domains, especially for performing large-scale numerical computations on large data. This talk explains you how to use the library to train machine-learning models and make your next application smarter.

Speaker:

Mário Cordeiro
Universidade do Porto, Porto, Portugal
Mário

Transcript

1. Dive deep into Machine Learning with . Mário Cordeiro Checkmarx, Braga, 2018 2018-04-18 mario.miguel.cordeiro@gmail.com @mmfcordeiro https://www.pexels.com/photo/underwater-blue-ocean-sea-9149/
2. Dive deep into Machine Learning with . Mário Cordeiro Checkmarx, Braga, 2018 2018-04-18 mario.miguel.cordeiro@gmail.com @mmfcordeiro https://www.pexels.com/photo/underwater-blue-ocean-sea-9149/ with hands-on examples
3. In this presentation you will learn Introductory concepts of Machine Learning Few basic concepts of Deep Learning What is TensorFlow ▪ Key differences from other approaches ▪ the dataflow-like model ▪ APIs and High level APIs ▪ TensorBoard features
4. In this presentation you will see TensorFlow in action Ready to use Pre-trained models Transfer Learning techniques What happened in 2 years of TensorFlow Tutorials, Code labs and Videos
5. In the hands-on you will see 1. Linear Regression Normal Equation 2. Linear regression using batch gradient descent 3. Classification (Softmax Regression, or Multinomial Logistic Regression) 4. scikit-learn intro 5. Optical Character Recognition (simple logistic regression) 6. MNIST 1.0 (TensorFlow) with 1-layer neural network 7. MNIST 2.0 (TensorFlow) using 5 fully connected layers with a softmax 8. MNIST 3.0 (TensorFlow) using convolutional network 9. MNIST 4.0 (Keras + TensorFlow) 10. Inception Model and Transfer Learning
6. http://insights.dice.com/2016/09/26/deep-learning-core-technology/ What is
7. Machine Learning ~= Programming + Data y = f(x) What is Machine Learning? input prediction Properties: • Generic enough to deal with different domain problems • No specific domain rules encoded model
8. Example model for a Linear Regression y = W x + b What is Machine Learning? “Learn” means a data-driven procedure to find useful values for parameters input prediction function modeled by 2 parameters
9. Example model for a Linear Regression y = W x + b What is Machine Learning? input prediction slope
10. What is Machine Learning? What are we trying to do? Mystery equation: y = 0,1x + 0,3 + noise Model: y = Wx + b Objective: given enough (x, y) samples, figure out the value of W and b (x=1,y=3) (x=7,y=9) (x=2,y=5) (x=12,y=20) (x=9,y=11) (…) y=f(x) Data Model
11. What is Machine Learning? Let’s code this up Inference: f (x) { “given x, this is the code that predicts y” } Loss: loss (y, expected_y) {“quantify how bad our prediction was”} Optimize: update (W, b) {“update parameters W, b to improve the prediction”}
12. Hands-on 1. Linear Regression Normal Equation 2. Linear regression using batch gradient descent
13. What is Machine Learning? Pet Classifier y = W x + b input prediction parameters
14. What is Machine Learning? Finding the best features w c r w c r h h y = W x + b input prediction parameters x y are vectors, folks
15. Hands-on 3. Classification 4. scikit-learn intro
16. What is Machine Learning? Finding the best features w h c r w h c r def get_texture_feature ( pixels ): ...? I suspect this won’t work
17. Hands-on 5. Optical Character Recognition
18. What is Machine Learning? Problem: Many variations in same class!
19. What is Machine Learning? Problem: Many similarities between classes!
20. What is Machine Learning? Textures are Pixels def get_texture_feature ( pixels ): ...?
21. What is Machine Learning Deep Learning? No feature engineering needed Your lazy humans… just use my raw pixels!
22. What is Machine Learning Deep Learning? No feature engineering needed and Neural Networks
23. What is Deep Learning? Artificial Neural Network vs Deep Neural Network
24. What is Deep Learning? Artificial Neural Network vs Deep Neural Network pixels Feat Val size: 10 w: 7 h: 7 r: 2,5 c: features First Layers Learn the best features
25. What is Deep Learning? Playing the imitation game pixels + “dog” label Model predicted the Pet correctly = from a labeled Pet dataset shuffle and feed the DNN with Pets
26. What is Deep Learning? Playing the imitation game pixels + “parrot” label Model predicted the Pet incorrectly ≠ back propagate error and continue with training phase weights from a labeled Pet dataset shuffle and feed the DNN with Pets
27. What is Deep Learning? Playing the imitation game ▪ Loss function ▪ Optimization via Gradient Descend pixels + “parrot” label Model predicted the Pet incorrectly ≠ back propagate error and continue with training phase weights
28. What is Deep Learning? Playing the imitation game ▪ Loss function ▪ Optimization via Gradient Descend pixels + “cat” label Model predicted the Pet correctly = weights
29. What is Deep Learning? Playing the imitation game ▪ Loss function ▪ Optimization via Gradient Descend Parameter #1 Parameter #2 Loss
30. What is Deep Learning? http://www.businessinsider.com/what-is-google-tensorflow-2015-11
31. Hands-on 5. Optical Character Recognition (simple logistic regression) 6. MNIST 1.0 (TensorFlow) with 1-layer neural network 7. MNIST 2.0 (TensorFlow) using 5 fully connected layers with a softmax 8. MNIST 3.0 (TensorFlow) using convolutional network
32. TensorFlowPlayground http://playground.tensorflow.org
33. Why Deep Learning? Artificial Neural Networks (ANN) again? ▪ 50’s and 60’s ▪ Today’s Quality of Data + Quantity of Data + Computation Power ▪ Better training + faster training ▪ Allow us to build complex models + + Penn Treebank WebTreebank
34. Why Deep Learning? Artificial Neural Networks (ANN) again? ▪ 50’s and 60’s ▪ Today’s Quality of Data + Quantity of Data + Computation Power ▪ Better training + faster training ▪ Allow us to build complex models (Inception-v3) “trained from scratch on a desktop with 8 NVIDIA Tesla K40s in about 2 weeks” https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html 150,000 photographs 1000 object categories
35. Why Deep Learning? Current state-of-the-art in ▪ Image: classification, captioning ▪ Language: translation, parsing, summarization ▪ Speech: recognition, generation ▪ Games: AlphaGo, Atari https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html
36. https://events.linuxfoundation.org/sites/events/files/slides/Machine_Intelligence_at_Google_Scale-_Vision-Speech_API__TensorFlow_and_Cloud_ML.pdf
37. Growing use of Deep Learning at Google
38. Growing use of Deep Learning at Google
39. Growing use of Deep Learning at Google
40. Growing use of Deep Learning at Google
41. Growing use of Deep Learning at Google Web Driven Mobile Driven AI First
42. What is TensorFlow?
43. What is TensorFlow? TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms.
44. What is TensorFlow? A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems
45. What is TensorFlow? A graph of operations. A multidimensional array.
46. What is TensorFlow? Google Brain project ▪ DistBelief first-generation ▪ scalable distributed training and inference system ▪ TensorFlow is the second generation ▪ Apache 2.0 license Dataflow graph-based computation infrastructure ▪ General purpose (Great for Deep Learning) ▪ Distributed ▪ Parallel Heterogeneous devices ▪ Data Centers, CPUs, GPUs, TPUs and mobile devices
47. The dataflow-like model A graph is like a blueprint: ▪ defined using high-level language ▪ compiled and optimized ▪ executed (in parts or fully) on available devices ▪ nodes represent computations ▪ data (tensors) flows between them Tensor: ▪ a scalar is a tensor ▪ a vector is a tensor ▪ a matrix is a tensor ▪ a multidimensional array is a tensor
48. Ops: Kernels: TensorFlow architecture TensorFlow Distributed Execution System Add Mul Print Reshape … CPU GPU TPU Android iOS … C++ front-end ... Compound Ops Go Python front-end Layers Estimator Keras Canned Estimators High Level APIs • Models in a box • Train and evaluate models • Build models
49. Under the hood https://www.wired.com/2015/11/google-open-sourcing-tensorflow-shows-ais-future-is-data-not-code/
50. import tensorflow as tf b = tf.Variable(tf.zeros([100])) # 100-d vector, init to zeroes W = tf.Variable(tf.random_uniform([784,100],-1,1)) # 784x100 matrix w/rnd vals x = tf.placeholder(name="x") # Placeholder for input relu = tf.nn.relu(tf.matmul(W, x) + b) # Relu(Wx+b) C = [...] # Cost computed as a function # of Relu Code Fragment vs Computation Graph Computation Graph
51. # define a training step y_ = tf.placeholder(tf.float32, [None, 10]) xent = -tf.reduce_sum(y_*tf.log(y)) C = tf.train.GradientDescentOptimizer(0.01).minimize(xent) Code Fragment vs Training Step Computation Graph Built-in automatic gradient computation Parameter W Parameter b Loss
52. init = tf.initialize_all_variables() # initialize session s = tf.Session() s.run (init) for step in xrange(0, 10): input = ...construct 100-D input array ... # Create 100-d vector for input result = s.run(C, feed_dict={x: input}) # Fetch cost, feeding x=input print step, result Code Fragment vs Training Iterations Computation Graph Single machine or distributed system structure
53. Machine Learning using TensorFlow http://terencebroad.com/nnvis.html
54. Programming with TensorFlow Python API: Building graphs mostly like numpy ▪ With special operation for deep learning ▪ TF Learn like scikit-learn General Purpose Operations: ▪ Basics: constant, random, placeholder, cast, shape ▪ Variables: assign, assign_sub, assign_add ▪ Queues: enqueue, enqueue_batch, dequeue, blocking or not ▪ Logical: equal, greater, less, where, min, max, argmin, argmax ▪ Tensor computations: all math ops, matmul, determinant, inverse, cholesky ▪ Images: encode, decode, crop, pad, resize, color spaces, random perturbations ▪ Sparse tensors: represented as 3 tensors. ▪ File IO: file reader (sstable, recordio, txt, csv), parser protocol buffers ▪ Control flow: control dependencies, conditionals, loops, functions
55. Programming with TensorFlow Neural Net Specific Operations ▪ Activations: sigmoid, tanh, relu, relu6, elu, dropout ▪ Pooling: avg, max ▪ Convolutions: 1D, 2D, 3D with many options ▪ Normalization: local, batch, moving averages ▪ Classification: softmax, softmax loss, cross entropy loss, topk ▪ Embeddings: distributed lookups/gather, scatter/updates ▪ Sampling: candidate sampler (various options), sampling softmax ▪ Updates: "fused ops" to speed-up optimizer updates (Adagrad, Momentum.) ▪ Summaries: Capture information for visualization
56. Programming with TensorFlow High level APIs since r1.0 ▪ tf.layers: conv2d, conv3d, max_pooling1d, etc ▪ tf.estimator: LinearRegressor, LinearClassifier, DNNRegressor ▪ tf.keras: models, estimators, optimizers, etc ▪ Canned Estimators: cnn, LTSM, etc # Instantiate a Keras inception v3 model. keras_inception_v3 = tf.keras.applications.inception_v3.InceptionV3(weights=None) # Compile model with the optimizer, loss, and metrics you'd like to train with. keras_inception_v3.compile(optimizer=tf.keras.optimizers.SGD(lr=0.0001, momentum=0.9), loss='categorical_crossentropy', metric='accuracy') # Create an Estimator from the compiled Keras model. est_inception_v3 = tf.keras.estimator.model_to_estimator(keras_model=keras_inception_v3) # Treat the derived Estimator as you would any other Estimator. For example, # the following derived Estimator calls the train method: est_inception_v3.train(input_fn=my_training_set, steps=2000)
57. Programming with TensorFlow High level APIs since r1.0 ▪ tf.layers: conv2d, conv3d, max_pooling1d, etc ▪ tf.estimator: LinearRegressor, LinearClassifier, DNNRegressor ▪ tf.keras: models, estimators, optimizers, etc ▪ Canned Estimators: cnn, LTSM, etc Best practices and default settings included
58. Hands-on 9. MNIST 4.0 (Keras + TensorFlow)
59. TensorBoard Interactive Computation Graph Visualization
60. TensorBoard Visualizing Learning http://localhost:6006 tensorboard --logdir /tmp/retrain_logs
61. TensorBoard Embedding Visualization for high dimensional data
62. TensorFlow in action
63. Android MNIST Detect handwritten digits in Android with Tensorflow https://github.com/miyosuda/TensorFlowAndroidMNIST Accuracy: ▪ Linear ~90% ▪ Two-layer DNN ~96% ▪ ConvNet ~99% (close to the state of the art on this toy problem)
64. From Pixels to Actions Human-level control through Deep Reinforcement Learning https://research.googleblog.com/2015/02/from-pixels-to-actions-human-level.html https://github.com/devsisters/DQN-tensorflow
65. From Pixels to Actions “Deep Reinforcement Learning: Playing a Racing Game” by Pedro Lopes https://lopespm.github.io/machine_learning/2016/10/06/deep-reinforcement-learning-racing-game.html
66. AlphaGo Mastering the ancient game of Go with Machine Learning https://research.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html
67. lip-read Google’s DeepMind AI can lip-read TV shows better than a pro https://www.newscientist.com/article/2113299-googles-deepmind-ai-can-lip-read-tv-shows-better-than-a-pro/ “Our implementation is based on the TensorFlow library and trained on a GeForce Titan X GPU with 12GB memory. […] The model was trained for around 500,000 iterations, which took approximately 10 days.” arXiv, DOI: arXiv:1611.05358v1
68. Universal language translation Google's AI can translate language pairs it has never seen https://www.engadget.com/2016/11/24/google-ai-translate-language-pairs-it-has-never-seen/ “All training is carried out as in our general NMT pipeline as described in [24] and implemented in TensorFlow.” arXiv, DOI: arXiv:1611.04558v1
69. Deep Koalarization Image colorization using deep CNN and features extracted from Inception-ResNet-v2 https://github.com/baldassarreFe/deep-koalarization
70. EnhanceNet EnhanceNet: Single Image Super-Resolution Through Automated Texture Synthesis https://arxiv.org/pdf/1612.07919.pdf input image highest PSNR ENet-PAT
71. Create your own artwork Deep Dream, Style Transfer and Inceptionism https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html https://deepdreamgenerator.com/ https://www.instagram.com/deepdreamgenerator/ http://ostagram.ru/static_pages/lenta?last_days=30
72. Pre-trained Models
73. Inception Pre-trained model for Image Classification https://research.googleblog.com/2016/08/improving-inception-and-image.html An Alaskan Malamute (left) and a Siberian Husky (right). Images from Wikipedia
74. Show and Tell Pre-trained model for Image Captioning https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html Model generates a completely new caption using concepts learned from similar scenes in the training set Automatically captioned by the system
75. ParseyMcParseface Pre-trained models for language parsing (NLP part-of-speech tagging) https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html
76. Text summarization with TensorFlow Pre-trained model for Extractive and Abstractive summarization https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html Original Text: Alice and Bob took the train to visit the zoo. They saw a baby giraffe, a lion, and a flock of colorful tropical birds. Extractive Summary: Alice and Bob visit the zoo. saw a flock of birds. Abstractive summary: Alice and Bob visited the zoo and saw animals and birds
77. Transfer Learning Extending pre-trained models for your classes https://codelabs.developers.google.com/codelabs/tensorflow-for-poets https://github.com/Hvass-Labs/TensorFlow-Tutorials/blob/master/08_Transfer_Learning.ipynb Training Inception v3 takes 2 weeks in 8 NVIDIA Tesla K40s https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html Building the Transfer Values and training just the Fully- Connected Layer takes seconds in regular PC without GPU.
78. Hands-on 10. Inception Model and Transfer Learning
79. 2 Years of TensorFlow
80. r0.5 r0.6 r0.7 r0.8 r0.9 r0.10 r0.11 r0.12 r1.0 r1.1 r1.2 r1.3 r1.4 2 Years of TensorFlow Initial Release TensorFlow Serving Python 3.3 Faster on GPUs CuDNN v5 CUDA 8 HDFS XLA Java TF Debugger 58x speedup Windows Go API TensorBoard Embeddings Distributed (multi-machine) based on gRPC iOS support Raspberry Pi (IoT) TF-Slim (tf.contrib.slim) Java API for Windows Keras 2 API to contrib Python 3.6 on Windows Canned estimators • DNNClassifier • DNNRegressor • LinearClassifier • LinearRegressor cuDNN v6 tf.keras and tf.data part of core TensorFlow API Nov ‘15 Nov ‘16 Nov ‘17 Feb ‘17
81. 2 Years of TensorFlow Community: ▪ Most popular Machine Learning project on GitHub http://leotam.github.io/general/2017/04/10/Frameworks.html
82. 2 Years of TensorFlow https://twitter.com/fchollet/status/915626952408436736
83. https://www.pexels.com/photo/black-and-white-lake-dive-header-4465/ Ready to dive into TensorFlow?
84. Ready to dive into TensorFlow? Heterogeneous devices Same API ▪ for Researchers, Data Scientists and Developers ▪ from Design to Production Use/Adapt state-of-the-art models Great ecosystem, excellent community support, open source Easy model deployment to devices and Cloud ML Deeplearning4j vs PyTorch vs TensorFlow vs Caffe vs Keras vs MxNet vs Gluon & CNTK https://deeplearning4j.org/compare-dl4j-tensorflow-pytorch
85. Ready to dive into TensorFlow? TensorFlow white papers http://download.tensorflow.org/paper/whitepaper2015.pdf https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf TensorFlow Tutorials https://www.tensorflow.org/tutorials Machine Learning Recipes with Josh Gordon [beginner] https://www.youtube.com/watch?v=Gj0iyo265bc&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal Magnus Erik Hvass Pedersen TensorFlow Tutorials [intermediate] https://github.com/Hvass-Labs/TensorFlow-Tutorials https://www.youtube.com/playlist?list=PL9Hr9sNUjfsmEu1ZniY0XpHSzl5uihcXZ TensorFlow Dev Summit 2017 Keynote and Sessions https://events.withgoogle.com/tensorflow-dev-summit/videos-and-agenda/#content
86. TensorFlow for Poets Codelab You will learn: ▪ Transfer Learning in images You will need: ▪ your laptop (linux, macOS, windows) ▪ Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2/ or 16 Minutes 45 Minutes
87. TensorFlow and deep learning, without a PhD You will learn: ▪ recognize handwritten digits (MNIST) You will need: ▪ your laptop (linux, macOS, windows) ▪ Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/cloud-tensorflow-mnist or 149 Minutes
88. Data Science Community in Portugal https://www.meetup.com/datascienceportugal https://www.facebook.com/datascienceportugal https://datascienceportugal.slack.com
89. Thank you @mmfcordeiro mario.miguel.cordeiro@gmail.com https://www.pexels.com/photo/black-and-white-lake-dive-header-4465/
90. Extras
92. More TensorFlow in action
93. Create your own artwork Fast Style Transfer in TensorFlow https://github.com/lengstrom/fast-style-transfer
94. Welcome to Magenta! Machine intelligence for music and art generation https://magenta.tensorflow.org/welcome-to-magenta
95. Google Cloud ML Platform
96. Google Cloud ML Platform Managed Scalable Machine Learning Platform ▪ Prediction at Scale ▪ Build Machine Learning Models Easily ▪ Fully Managed Platform ▪ Power by TensorFlow APIs ▪ Image Analysis ▪ Speech Recognition ▪ Translation
97. Mainstream Machine Learning Cloud ML and APIs https://cloud.google.com/products/machine-learning/
98. Models into Production TensorFlow Serving https://tensorflow.github.io/serving/