Dive deep into machine learning with TensorFlow @ GDG Porto Devfest 2016

Dive deep into machine learning with TensorFlow is a high-level introduction to the TensorFlow library, architecture and API. Originally developed at Google for the purposes of conducting machine-learning and deep neural networks research, TensorFlow leverages a general computational model that is applicable in a wide variety of other domains, especially for performing large-scale numerical computations on large data. This talk explains you how to use the library to train machine-learning models and make your next application smarter.

Speakers

Mário Cordeiro
Universidade do Porto, Porto, Portugal
Mário

Transcript

1. Dive deep into machine learning with . Mário Cordeiro Porto GDG DevFest 2016 2016-12-02 mario.miguel.cordeiro@gmail.com @mmfcordeiro https://www.pexels.com/photo/underwater-blue-ocean-sea-9149/
2. In this presentation you will learn Basic of ML and Deep Learning What is TensorFlow, the dataflow-like model, API and TensorBoard TensorFlow in action, please! Pre-trained models and Learning Transfer To evaluate the right deep learning library for you
3. http://insights.dice.com/2016/09/26/deep-learning-core-technology/ What is
4. Machine Learning ~= Programming + Data y = W x + b What is Machine Learning? “Learn” means a data-driven procedure to find useful values for parameters input prediction parameters
5. Linear Regression y = W x + b What is Machine Learning? input prediction slope
6. What is Machine Learning? What are we trying to do? Mystery equation: y = 0,1x + 0,3 + noise Model: y = Wx + b Objective: given enough (x, y) samples, figure out the value of W and b
7. What is Machine Learning? Let’s code this up Inference: f (x) { “given x, this is the code that predicts y” } Loss: loss (y, expected_y) {“quantify how bad our prediction was”} Optimize: update (W, b) {“update parameters W, b to improve the prediction”}
8. What is Machine Learning? Fruit Classifier
9. What is Machine Learning? Finding the best features w h c r w h c r
10. What is Machine Learning? Finding the best features w h c r w h c r def get_texture_feature ( pixels ): ...?
11. What is Machine Learning? Too many variations!
12. What is Machine Learning? Textures are Pixels
13. What is Machine Learning Deep Learning? No feature engineering needed What!? Neural Networks just use my raw pixels
14. What is Deep Learning? Artificial Neural Network vs Deep Neural Network
15. What is Deep Learning? Playing the imitation game pixels Feat Val size: 10 w: 7 h: 7 r: 2,5 c: features
16. What is Deep Learning? Playing the imitation game pixels + “orange” label Model predicted the fruit correctly = from a labeled fruit dataset shuffle and feed the DNN with fruits
17. What is Deep Learning? Playing the imitation game pixels + “pineapple” label Model predicted the fruit incorrectly ≠ Back propagate error and continue weights from a labeled fruit dataset shuffle and feed the DNN with fruits
18. What is Deep Learning? Playing the imitation game  Loss function  Optimization via Gradient Descend pixels + “pineapple” label Model predicted the fruit incorrectly ≠ Back propagate error and continue weights
19. What is Deep Learning? Playing the imitation game  Loss function  Optimization via Gradient Descend pixels + “apple” label Model predicted the fruit correctly = weights
20. What is Deep Learning? Playing the imitation game  Loss function  Optimization via Gradient Descend Parameter #1 Parameter #2 Loss
21. What is Deep Learning? http://www.businessinsider.com/what-is-google-tensorflow-2015-11
22. TensorFlowPlayground http://playground.tensorflow.org
23. Why Deep Learning? Artificial Neural Networks (ANN) again?  50’s and 60’s  Today’s Quality of Data + Quantity of Data + Computation Power  Better training + faster training  Allow us to build complex models + + Penn Treebank WebTreebank
24. Why Deep Learning? Artificial Neural Networks (ANN) again?  50’s and 60’s  Today’s Quality of Data + Quantity of Data + Computation Power  Better training + faster training  Allow us to build complex models “trained from scratch on a desktop with 8 NVIDIA Tesla K40s in about 2 weeks” https://research.googleblog.com/2016/03/train-your-own-image-classifier-with.html 150,000 photographs 1000 object categories
25. Why Deep Learning? Current state-of-the-art in  Image: classification, captioning  Language: translation, parsing, summarization  Speech: recognition, generation  Games: AlphaGo, Atari
26. https://events.linuxfoundation.org/sites/events/files/slides/Machine_Intelligence_at_Google_Scale-_Vision-Speech_API__TensorFlow_and_Cloud_ML.pdf
27. What is TensorFlow?
28. What is TensorFlow? TensorFlow is an interface for expressing machine learning algorithms, and an implementation for executing such algorithms.
29. What is TensorFlow? A computation expressed using TensorFlow can be executed with little or no change on a wide variety of heterogeneous systems
30. What is TensorFlow? A graph of operations. A multidimensional array.
31. What is TensorFlow? Google Brain project  DistBelief first-generation  scalable distributed training and inference system  TensorFlow is the second generation  Apache 2.0 license Dataflow graph-based computation infrastructure  General purpose (Great for Deep Learning)  Distributed  Parallel Heterogeneous devices  Data Centers, CPUs, GPUs, TPUs and mobile devices
32. The dataflow-like model A graph is like a blueprint:  defined using high-level language  compiled and optimized  executed (in parts or fully) on available devices  nodes represent computations  data (tensors) flows between them Tensor:  a scalar is a tensor  a vector is a tensor  a matrix is a tensor  a multidimensional array is a tensor
33. Ops: Kernels: TensorFlow architecture TensorFlow Distributed Execution System TensorFlow Distributed Execution System Add Add Mul Mul Print Print Reshape Reshape … … CPU CPU GPU GPU TPU TPU Android Android iOS iOS … … C++ front-end C++ front-end Python front-end Python front-end ... Compound Ops
34. Under the hood https://www.wired.com/2015/11/google-open-sourcing-tensorflow-shows-ais-future-is-data-not-code/
35. import tensorflow as tf b = tf.Variable(tf.zeros([100])) # 100-d vector, init to zeroes W = tf.Variable(tf.random_uniform([784,100],-1,1)) # 784x100 matrix w/rnd vals x = tf.placeholder(name="x") # Placeholder for input relu = tf.nn.relu(tf.matmul(W, x) + b) # Relu(Wx+b) C = [...] # Cost computed as a function # of Relu Code Fragment vs Computation Graph Computation Graph
36. # define a training step y_ = tf.placeholder(tf.float32, [None, 10]) xent = -tf.reduce_sum(y_*tf.log(y)) C = tf.train.GradientDescentOptimizer(0.01).minimize(xent) Code Fragment vs Training Step Computation Graph Built-in automatic gradient computation Parameter W Parameter b Loss
37. init = tf.initialize_all_variables() # initialize session s = tf.Session() s.run (init) for step in xrange(0, 10): input = ...construct 100-D input array ... # Create 100-d vector for input result = s.run(C, feed_dict={x: input}) # Fetch cost, feeding x=input print step, result Code Fragment vs Training Iterations Computation Graph Single machine or distributed system structure
38. Machine Learning using TensorFlow http://terencebroad.com/nnvis.html
39. Programming with TensorFlow Python API: Building graphs mostly like numpy  With special operation for deep learning  TF Learn like scikit-learn General Purpose Operations:  Basics: constant, random, placeholder, cast, shape  Variables: assign, assign_sub, assign_add  Queues: enqueue, enqueue_batch, dequeue, blocking or not  Logical: equal, greater, less, where, min, max, argmin, argmax  Tensor computations: all math ops, matmul, determinant, inverse, cholesky  Images: encode, decode, crop, pad, resize, color spaces, random perturbations  Sparse tensors: represented as 3 tensors.  File IO: file reader (sstable, recordio, txt, csv), parser protocol buffers  Control flow: control dependencies, conditionals, loops, functions
40. Programming with TensorFlow Neural Net Operations  Activations: sigmoid, tanh, relu, relu6, elu, dropout  Pooling: avg, max  Convolutions: 1D, 2D, 3D with many options  Normalization: local, batch, moving averages  Classification: softmax, softmax loss, cross entropy loss, topk  Embeddings: distributed lookups/gather, scatter/updates  Sampling: candidate sampler (various options), sampling softmax  Updates: "fused ops" to speed-up optimizer updates (Adagrad, Momentum.)  Summaries: Capture information for visualization
41. TensorBoard Visualizing Learning http://localhost:6006 tensorboard --logdir /tmp/retrain_logs
42. TensorBoard Interactive Computation Graph Visualization
43. TensorBoard Embedding Visualization for high dimensional data
44. TensorFlow in action TensorFlow in action
45. Android MNIST Detect handwritten digits in Android with Tensorflow https://github.com/miyosuda/TensorFlowAndroidMNIST Accuracy:  Linear ~90%  Two-layer DNN ~96%  ConvNet ~99% (close to the state of the art on this toy problem)
46. From Pixels to Actions Human-level control through Deep Reinforcement Learning https://research.googleblog.com/2015/02/from-pixels-to-actions-human-level.html https://github.com/devsisters/DQN-tensorflow
47. AlphaGo Mastering the ancient game of Go with Machine Learning https://research.googleblog.com/2016/01/alphago-mastering-ancient-game-of-go.html https://cloudplatform.googleblog.com/2016/05/Google-supercharges-machine-learning-tasks-with-custom-chip.html
48. lip-read Google’s DeepMind AI can lip-read TV shows better than a pro https://www.newscientist.com/article/2113299-googles-deepmind-ai-can-lip-read-tv-shows-better-than-a-pro/ “Our implementation is based on the TensorFlow library and trained on a GeForce Titan X GPU with 12GB memory. […] The model was trained for around 500,000 iterations, which took approximately 10 days.” arXiv, DOI: arXiv:1611.05358v1
49. Universal language translation Google's AI can translate language pairs it has never seen https://www.engadget.com/2016/11/24/google-ai-translate-language-pairs-it-has-never-seen/ “All training is carried out as in our general NMT pipeline as described in [24] and implemented in TensorFlow.” arXiv, DOI: arXiv:1611.04558v1
50. Create your own artwork Deep Dream, Style Transfer and Inceptionism https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html https://deepdreamgenerator.com/ https://www.instagram.com/deepdreamgenerator/ http://ostagram.ru/static_pages/lenta?last_days=30
51. Pre-trained Models
52. Inception Pre-trained model for Image Classification https://research.googleblog.com/2016/08/improving-inception-and-image.html An Alaskan Malamute (left) and a Siberian Husky (right). Images from Wikipedia
53. Show and Tell Pre-trained model for Image Captioning https://research.googleblog.com/2016/09/show-and-tell-image-captioning-open.html Model generates a completely new caption using concepts learned from similar scenes in the training set Automatically captioned by the system
54. ParseyMcParseface Pre-trained models for language parsing (NLP part-of-speech tagging) https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html
55. Text summarization with TensorFlow Pre-trained model for Extractive and Abstractive summarization https://research.googleblog.com/2016/08/text-summarization-with-tensorflow.html Original Text: Alice and Bob took the train to visit the zoo. They saw a baby giraffe, a lion, and a flock of colorful tropical birds. Extractive Summary: Alice and Bob visit the zoo. saw a flock of birds. Abstractive summary: Alice and Bob visited the zoo and saw animals and birds
56. TensorFlow’sFirst Year
57. A Year of TensorFlow Releases: r0.6: Python 3.3, Faster on GPUs r0.7: TensorFlow Serving r0.8: Distributed (multi-machine) support based on gRPC r0.9: iOS support, Raspberry Pi (IoT) r0.10: TF-Slim (tf.contrib.slim) r0.11: CuDNN v5, CUDA 8 support, HDFS Support
58. A Year of TensorFlow Community:  Most popular Machine Learning project on GitHub
59. A Year of TensorFlow http://www.kdnuggets.com/2016/11/top-20-python-machine-learning-open-source-updated.html
60. https://www.pexels.com/photo/black-and-white-lake-dive-header-4465/ Ready to dive into TensorFlow?
61. Ready to dive into TensorFlow? Heterogonous devices Same API for Researchers, Data Scientists and Developers Reuse state-of-the-art models Apart from being great ecosystem has excellent community support Easy model deployment to devices and Cloud ML Evaluation of Deep Learning Toolkits https://github.com/zer0n/deepframeworks/blob/master/README.md DL4J vs. Torch vs. Theano vs. Caffe vs. TensorFlow https://deeplearning4j.org/compare-dl4j-torch7-pylearn
62. Thank you @mmfcordeiro mario.miguel.cordeiro@gmail.com https://www.pexels.com/photo/black-and-white-lake-dive-header-4465/
63. Resources TensorFlow white paper http://download.tensorflow.org/paper/whitepaper2015.pdf TensorFlow Tutorials https://www.tensorflow.org/versions/r0.12/tutorials/index.html Machine Learning Recipes with Josh Gordon [beginner] https://www.youtube.com/watch?v=Gj0iyo265bc&list=PLOU2XLYxmsIIuiBfYad6rFYQU_jL2ryal Magnus Erik Hvass Pedersen TensorFlow Tutorials [intermediate] https://github.com/Hvass-Labs/TensorFlow-Tutorials
64. TensorFlow for Poets Codelab 15:15 @ Workshops Room You will need:  your laptop (linux, macOS, windows)  Docker or Docker Tools installed https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/
65. Extras
67. More TensorFlow in action More TensorFlow in action
68. Create your own artwork Fast Style Transfer in TensorFlow https://github.com/lengstrom/fast-style-transfer
69. Welcome to Magenta! Machine intelligence for music and art generation https://magenta.tensorflow.org/welcome-to-magenta
70. Google Cloud ML Platform
71. Google Cloud ML Platform Managed Scalable Machine Learning Platform  Prediction at Scale  Build Machine Learning Models Easily  Fully Managed Platform  Power by TensorFlow APIs  Image Analysis  Speech Recognition  Translation
72. Mainstream Machine Learning Cloud ML and APIs https://cloud.google.com/products/machine-learning/
73. Models into Production TensorFlow Serving https://tensorflow.github.io/serving/