$30 off During Our Annual Pro Sale. View Details »

Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

With TensorFlow.js developers can embed pre-trained deep learning models directly within a Node.js or web application. Deep learning can be used for many tasks including image or audio classification, facial recognition, object recognition, image caption generation, and natural language processing. TensorFlow.js enables inferencing on deep learning models at the edge, removing the dependency on potentially slow and unreliable network API calls.

Bradley Holt

March 15, 2019
Tweet

More Decks by Bradley Holt

Other Decks in Programming

Transcript

  1. Deep Learning
    at the Edge with
    TensorFlow.js
    Bradley Holt
    Program Manager, Developer Advocacy
    Center for Open-Source Data and AI Technologies
    @BradleyHolt
    DOC ID / Month XX, 2018 / © 2018 IBM Corporation

    View Slide

  2. Is AI overhyped?

    View Slide

  3. Yes

    View Slide

  4. Thank you
    for coming
    to my talk

    View Slide

  5. General AI
    Metal Skull With Terminator Eye by L.C. Nøttaasen, on Flickr (CC BY-SA 2.0).

    View Slide

  6. Broad AI
    -[ electrIc b88Gal88 ]- by JD Hancock, on Flickr (CC BY 2.0).

    View Slide

  7. Narrow AI
    Danbo on the Lookout by IQRemix, on Flickr (CC BY-SA 2.0).

    View Slide

  8. Deep
    Learning
    http://codait.org/

    View Slide

  9. Demo:
    Veremin,
    A Video
    Theremin
    Based on
    PoseNet
    https://veremin.mybluemix.net/

    View Slide

  10. View Slide

  11. Try Veremin Yourself
    https://veremin.mybluemix.net/

    View Slide

  12. Demo: Object
    Identification
    and Image
    Segmentation
    https://github.com/CODAIT/magicat

    View Slide

  13. https://github.com/CODAIT/magicat

    View Slide

  14. Install magicat
    $ npm install -g magicat
    + [email protected]
    added 255 packages from 209 contributors in 11.798s

    View Slide

  15. 23895621638_535be71dee_k.jpg
    37038669284_899d7784a9_k.jpg
    37489697170_31d05aa027_k.jpg
    37699459356_24fd526a5e_k.jpg
    37699976806_5ce694be36_k.jpg
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

    View Slide

  16. Scan Directory with magicat
    $ magicat .
    Scanning directory '/Users/bradleydholt/tfjs-demos/magicat'...
    The image '23895621638_535be71dee_k.jpg' contains the following
    segments: background, cat.
    The image '37038669284_899d7784a9_k.jpg' contains the following
    segments: background, sheep.
    The image '37489697170_31d05aa027_k.jpg' contains the following
    segments: background, sheep.
    The image '37699459356_24fd526a5e_k.jpg' contains the following
    segments: background, cat.
    The image '37699976806_5ce694be36_k.jpg' contains the following
    segments: background, horse.

    View Slide

  17. background, cat
    background, sheep
    background, sheep
    background, cat
    background, horse
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

    View Slide

  18. Save Image Segment with magicat
    $ magicat 37699976806_5ce694be36_k.jpg --save horse
    The image '37699976806_5ce694be36_k.jpg' contains the following
    segments: background, horse.
    saved 37699976806_5ce694be36_k-horse.png
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

    View Slide

  19. Demo: Object
    Identification
    and Image
    Segmentation
    with the Magic
    Cropping Tool
    https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/

    View Slide

  20. https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/

    View Slide

  21. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480

    View Slide

  22. Deep Learning Use Cases
    https://ibm.biz/max-developers

    View Slide

  23. Software
    Programming
    Untitled by Marcin Wichary, on Flickr (CC BY 2.0).

    View Slide

  24. 0101100100101100000110101101
    0101001001100000110101101011
    0110010010101010011101001011
    0101010011100101010101010000
    1010101011100000011010101010
    1001000110000011010011010101
    0010001011010101011101000101
    Software Program
    Business Logic

    View Slide

  25. 0101100100101100000110101101
    0101001001100000110101101011
    0110010010101010011101001011
    0101010011100101010101010000
    1010101011100000011010101010
    1001000110000011010011010101
    0010001011010101011101000101
    Software Program
    Input
    Program Execution
    Output

    View Slide

  26. Deep
    Learning

    View Slide

  27. Input Layer Hidden Layers Output Layer

    View Slide

  28. Labeled Training Data
    Backpropagation
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).
    Output Errors
    Cat

    Not Cat

    Cat ❌
    Cat
    Cat

    View Slide

  29. Input
    Neural Network Inferencing
    Output
    Cat
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

    View Slide

  30. Neural Network Inferencing
    Output
    Not Cat
    Input
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

    View Slide

  31. Machine
    Learning
    Libraries
    The Leeds Library by Michael D Beckwith, on Flickr (CC0 1.0).

    View Slide

  32. View Slide

  33. View Slide

  34. Demo: Minimal
    Train and
    Predict with
    TensorFlow.js
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  35. Training Data (y = 2x - 1)
    x y
    -1 -3
    0 -1
    1 1
    2 3
    3 5
    4 7
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  36. Prepare the Model for Training
    // Create a sequential model
    const model = tf.sequential();
    model.add(tf.layers.dense({units: 1, inputShape: [1]}));
    // Prepare the model for training
    model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  37. Train the Model
    // Create synthetic training data (y = 2x - 1)
    const xs = tf.tensor2d([-1, 0, 1, 2, 3, 4], [6, 1]);
    const ys = tf.tensor2d([-3, -1, 1, 3, 5, 7], [6, 1]);
    // Train the model
    await model.fit(xs, ys, {epochs: 250});
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  38. Predict Outputs on Training Inputs
    // Predict outputs for training inputs
    const trainingInputs = [-1, 0, 1, 2, 3, 4];
    const trainingOutputs = trainingInputs.map(input =>
    model.predict(tf.tensor2d([input], [1, 1]))
    );
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  39. Sample Prediction Results on
    Training Data
    x y inferenced y calculated
    -1 -2.8576648235321045 -3
    0 -0.892378032207489 -1
    1 1.072908639907837 1
    2 3.0381953716278076 3
    3 5.003482341766357 5
    4 6.968769073486328 7
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  40. Predict Outputs on Test Inputs
    // Predict outputs for a set of test inputs
    const testInputs = [3, 14, 25, 43, 56, 72];
    const testOutputs = testInputs.map(input =>
    model.predict(tf.tensor2d([input], [1, 1]))
    );
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  41. Sample Prediction Results on
    Test Data
    x y inferenced y calculated
    3 5.003482341766357 5
    14 26.62163734436035 27
    25 48.23978805541992 49
    43 83.61495208740234 85
    56 109.16368103027344 111
    72 140.60826110839844 143
    https://github.com/bradley-holt/tfjs-demos

    View Slide

  42. Applying
    Deep
    Learning
    Data Science
    Expertise
    Computing
    Resources
    High-Quality
    Training Data
    Model Deployment
    Time
    Model Integration
    Inferencing Code
    And more…

    View Slide

  43. Model Asset Exchange (MAX)
    Classify
    Generate
    Recognize
    https://developer.ibm.com/exchanges/models/

    View Slide

  44. Choose deployable model
    Deep Learning asset on MAX
    Deploy
    Swagger specification Inference endpoint Metadata endpoint
    Microservice
    Input pre-processing, model execution, and output post-processing
    Deploy
    model
    Use
    model
    https://developer.ibm.com/exchanges/models/

    View Slide

  45. Deep
    Learning
    at the
    Edge
    Global Network by fdecomite, on Flickr (CC BY 2.0).

    View Slide

  46. Device
    Deep Learning
    Model
    Data
    Cloud Results
    Results
    Network

    View Slide

  47. Device
    Deep Learning
    Model
    Data
    Results

    View Slide

  48. The
    Cloud
    STOP
    Doggy Driver by Eric Sonstroem, on Flickr (CC BY 2.0).

    View Slide

  49. Edge Device Inferencing*
    A large model with large input or
    output requires a slow and
    unreliable network API call
    July 13, 2018 / © 2018 IBM Corporation
    Large Model
    *Assumes limited edge device capabilities
    Fast
    Inferencing
    A network API request with small
    input and output can achieve fast,
    but unreliable, inferencing
    A small model run locally can
    achieve fast and reliable inferencing
    with large input and output
    Large
    Input/Output

    View Slide

  50. Converting a
    Pre-Trained
    Model to
    TensorFlow.js
    Portage by ThreeIfByBike, on Flickr (CC BY-SA 2.0).

    View Slide

  51. https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c

    View Slide

  52. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/

    View Slide

  53. Downloading Model Files
    deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
    http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz

    View Slide

  54. Extracting Model Files
    $ tar xzf deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
    $ ls -lh deeplabv3_mnv2_pascal_trainval
    8.4M frozen_inference_graph.pb
    16M model.ckpt-30000.data-00000-of-00001
    18K 2018 model.ckpt-30000.index

    View Slide

  55. TensorFlow Model Formats
    Frozen Graphs: Encapsulates model data in a single file
    Checkpoints: Allows for resuming of training
    SavedModel: TensorFlow’s Universal serialization format
    HDF5: Used by Keras to store model data

    View Slide

  56. Exploring the Model
    https://github.com/lutzroeder/Netron

    View Slide

  57. Model Input and Output

    View Slide

  58. Model Input and Output

    View Slide

  59. Model Input and Output

    View Slide

  60. Model Input and Output

    View Slide

  61. Model Input and Output

    View Slide

  62. Installing the TensorFlow.js Converter
    $ pip install tensorflowjs

    View Slide

  63. Converting the Model
    $ tensorflowjs_converter \
    --input_format=tf_frozen_model \
    --output_node_names='SemanticPredictions' \
    frozen_inference_graph.pb \
    model
    $ ls -lh model
    4.0M group1-shard1of3
    4.0M group1-shard2of3
    27K group1-shard3of3
    63K tensorflowjs_model.pb
    16K weights_manifest.json

    View Slide

  64. Image Segmenter Demo App
    https://github.com/vabarbosa/tfjs-sandbox

    View Slide

  65. Including the TensorFlow.js Library

    <br/>

    View Slide

  66. Loading the Model in TensorFlow.js
    const MODEL_URL = '/model/tensorflowjs_model.pb'
    const WEIGHTS_URL = '/model/weights_manifest.json'
    const model = await tf.loadFrozenModel(MODEL_URL, WEIGHTS_URL)

    View Slide

  67. Pre-Processing Model Input
    // create a tensor from an image
    const imageTensor = tf.fromPixels(imageElement)
    // insert a dimension into the tensor's shape
    const preprocessedInput = imageTensor.expandDims()

    View Slide

  68. Running Predictions with the Model
    // generates output prediction
    const prediction = model.predict(preprocessedInput)

    View Slide

  69. Post-Processing Model Output
    // get image width, height from output
    const imageWidth = prediction.shape[2]
    const imageHeight = prediction.shape[1]
    // create a regular array from the model's output prediction (tensor)
    const segMap = Array.from(prediction.dataSync())
    // create a new array with the corresponding segmentation colors
    const segMapColor = segMap.map(seg => colorMap[seg])

    View Slide

  70. Image Segmenter Demo App
    https://github.com/vabarbosa/tfjs-sandbox

    View Slide

  71. brain.js ONNX.js
    https://github.com/BrainJS/brain.js | https://github.com/Microsoft/onnxjs
    Other JavaScript ML Libraries

    View Slide

  72. What will you build
    with deep learning
    at the edge?
    AZ 89A by Kevin Dooley, on Flickr (CC BY 2.0).

    View Slide

  73. Resources: Demos
    • Veremin: A Video Theremin Based on PoseNet
    https://veremin.mybluemix.net/
    • Veremin—A Browser-based Video Theremin
    Making music visually using TensorFlow.js, PoseNet, and the Web MIDI & Web Audio APIs
    https://medium.com/ibm-watson-data-lab/veremin-a-browser-based-video-theremin-1548b63200c
    • !" magicat
    https://github.com/CODAIT/magicat
    • An Ode to the Terminal: The (Almost) Perfect Tool
    Bringing Deep Learning to the Terminal with Node and TensorFlow.js
    https://medium.com/ibm-watson-data-lab/an-ode-to-the-terminal-the-almost-perfect-tool-3f4e8435b6b1
    • Deploy a deep learning-powered ‘Magic cropping tool’
    Deploy a deep learning-powered 'Magic cropping tool' using pretrained open source models
    https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/

    View Slide

  74. Resources: Demos (cont’d)
    • Taking Selfies (and More) to the Next Level with Open Source Deep Learning Models
    Everything I’ve Learned About Machine Learning I’ve Learned From Machines.
    https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480
    • TensorFlow.js Demos
    https://github.com/bradley-holt/tfjs-demos
    • Bring Machine Learning to the Browser With TensorFlow.js — Parts I, II & III
    https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c
    https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-ii-7555ed9a999e
    https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3
    • TensorFlow.js Model Playground
    Pre-trained models converted to the TensorFlow.js web friendly format
    https://github.com/vabarbosa/tfjs-sandbox

    View Slide

  75. Resources: Machine Learning
    • TensorFlow
    https://www.tensorflow.org/
    • Keras
    https://keras.io/
    • PyTorch
    https://pytorch.org/
    • Caffe2
    https://caffe2.ai/
    • ONNX
    https://onnx.ai/
    • Netron
    Visualizer for deep learning and machine learning models
    https://github.com/lutzroeder/Netron

    View Slide

  76. Resources: Edge Machine Learning
    • TensorFlow.js
    https://js.tensorflow.org/
    • TensorFlow Lite
    https://www.tensorflow.org/lite
    • ONNX.js
    https://github.com/Microsoft/onnxjs
    • brain.js
    ! Neural networks in JavaScript
    https://github.com/BrainJS/brain.js
    • Core ML
    https://developer.apple.com/machine-learning/

    View Slide

  77. Resources: IBM Developer
    • Center for Open-Source Data & AI Technologies (CODAIT)
    http://codait.org/
    • IBM Developer Model Asset Exchange (MAX)
    https://developer.ibm.com/exchanges/models/
    • Model Asset Exchange (MAX) Code Patterns
    https://ibm.biz/max-developers
    • Image Segmenter
    Identify objects in an image, additionally assigning each pixel of the image to a particular object.
    https://developer.ibm.com/exchanges/models/all/max-image-segmenter/
    • Image Segmenter Model Files
    http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz

    View Slide

  78. Resources: IBM Cloud
    • IBM Cloud
    https://ibm.biz/Bd2A5f
    • IBM Watson Studio
    https://www.ibm.com/cloud/watson-studio
    • IBM Watson Machine Learning
    https://www.ibm.com/cloud/machine-learning
    • IBM Watson OpenScale
    https://www.ibm.com/cloud/watson-openscale

    View Slide

  79. Thank you
    codait.org
    twitter.com/codait_datalab
    medium.com/ibm-watson-data-lab
    github.com/codait
    developer.ibm.com

    View Slide

  80. View Slide