Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

With TensorFlow.js developers can embed pre-trained deep learning models directly within a Node.js or web application. Deep learning can be used for many tasks including image or audio classification, facial recognition, object recognition, image caption generation, and natural language processing. TensorFlow.js enables inferencing on deep learning models at the edge, removing the dependency on potentially slow and unreliable network API calls.

Transcript

1. Deep Learning
   at the Edge with
   TensorFlow.js
   Bradley Holt
   Program Manager, Developer Advocacy
   Center for Open-Source Data and AI Technologies
   @BradleyHolt
   DOC ID / Month XX, 2018 / © 2018 IBM Corporation
   

   View Slide

2. Is AI overhyped?
   

   View Slide

3. Yes
   

   View Slide

4. Thank you
   for coming
   to my talk
   

   View Slide

5. General AI
   Metal Skull With Terminator Eye by L.C. Nøttaasen, on Flickr (CC BY-SA 2.0).
   

   View Slide

6. Broad AI
   -[ electrIc b88Gal88 ]- by JD Hancock, on Flickr (CC BY 2.0).
   

   View Slide

7. Narrow AI
   Danbo on the Lookout by IQRemix, on Flickr (CC BY-SA 2.0).
   

   View Slide

8. Deep
   Learning
   http://codait.org/
   

   View Slide

9. Demo:
   Veremin,
   A Video
   Theremin
   Based on
   PoseNet
   https://veremin.mybluemix.net/
   

   View Slide

10. View Slide

11. Try Veremin Yourself
    https://veremin.mybluemix.net/
    

    View Slide

12. Demo: Object
    Identification
    and Image
    Segmentation
    https://github.com/CODAIT/magicat
    

    View Slide

13. https://github.com/CODAIT/magicat
    

    View Slide

14. Install magicat
    $ npm install -g magicat
    + magicat@1.4.9
    added 255 packages from 209 contributors in 11.798s
    

    View Slide

15. 23895621638_535be71dee_k.jpg
    37038669284_899d7784a9_k.jpg
    37489697170_31d05aa027_k.jpg
    37699459356_24fd526a5e_k.jpg
    37699976806_5ce694be36_k.jpg
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).
    

    View Slide

16. Scan Directory with magicat
    $ magicat .
    Scanning directory '/Users/bradleydholt/tfjs-demos/magicat'...
    The image '23895621638_535be71dee_k.jpg' contains the following
    segments: background, cat.
    The image '37038669284_899d7784a9_k.jpg' contains the following
    segments: background, sheep.
    The image '37489697170_31d05aa027_k.jpg' contains the following
    segments: background, sheep.
    The image '37699459356_24fd526a5e_k.jpg' contains the following
    segments: background, cat.
    The image '37699976806_5ce694be36_k.jpg' contains the following
    segments: background, horse.
    

    View Slide

17. background, cat
    background, sheep
    background, sheep
    background, cat
    background, horse
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).
    

    View Slide

18. Save Image Segment with magicat
    $ magicat 37699976806_5ce694be36_k.jpg --save horse
    The image '37699976806_5ce694be36_k.jpg' contains the following
    segments: background, horse.
    saved 37699976806_5ce694be36_k-horse.png
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).
    

    View Slide

19. Demo: Object
    Identification
    and Image
    Segmentation
    with the Magic
    Cropping Tool
    https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
    

    View Slide

20. https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
    

    View Slide

21. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480
    

    View Slide

22. Deep Learning Use Cases
    https://ibm.biz/max-developers
    

    View Slide

23. Software
    Programming
    Untitled by Marcin Wichary, on Flickr (CC BY 2.0).
    

    View Slide

24. 0101100100101100000110101101
    0101001001100000110101101011
    0110010010101010011101001011
    0101010011100101010101010000
    1010101011100000011010101010
    1001000110000011010011010101
    0010001011010101011101000101
    Software Program
    Business Logic
    

    View Slide

25. 0101100100101100000110101101
    0101001001100000110101101011
    0110010010101010011101001011
    0101010011100101010101010000
    1010101011100000011010101010
    1001000110000011010011010101
    0010001011010101011101000101
    Software Program
    Input
    Program Execution
    Output
    

    View Slide

26. Deep
    Learning
    

    View Slide

27. Input Layer Hidden Layers Output Layer
    

    View Slide

28. Labeled Training Data
    Backpropagation
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).
    Output Errors
    Cat
    ❌
    Not Cat
    ❌
    Cat ❌
    Cat
    Cat
    

    View Slide

29. Input
    Neural Network Inferencing
    Output
    Cat
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).
    

    View Slide

30. Neural Network Inferencing
    Output
    Not Cat
    Input
    Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).
    

    View Slide

31. Machine
    Learning
    Libraries
    The Leeds Library by Michael D Beckwith, on Flickr (CC0 1.0).
    

    View Slide

32. View Slide

33. View Slide

34. Demo: Minimal
    Train and
    Predict with
    TensorFlow.js
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

35. Training Data (y = 2x - 1)
    x y
    -1 -3
    0 -1
    1 1
    2 3
    3 5
    4 7
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

36. Prepare the Model for Training
    // Create a sequential model
    const model = tf.sequential();
    model.add(tf.layers.dense({units: 1, inputShape: [1]}));
    // Prepare the model for training
    model.compile({loss: 'meanSquaredError', optimizer: 'sgd'});
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

37. Train the Model
    // Create synthetic training data (y = 2x - 1)
    const xs = tf.tensor2d([-1, 0, 1, 2, 3, 4], [6, 1]);
    const ys = tf.tensor2d([-3, -1, 1, 3, 5, 7], [6, 1]);
    // Train the model
    await model.fit(xs, ys, {epochs: 250});
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

38. Predict Outputs on Training Inputs
    // Predict outputs for training inputs
    const trainingInputs = [-1, 0, 1, 2, 3, 4];
    const trainingOutputs = trainingInputs.map(input =>
    model.predict(tf.tensor2d([input], [1, 1]))
    );
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

39. Sample Prediction Results on
    Training Data
    x y inferenced y calculated
    -1 -2.8576648235321045 -3
    0 -0.892378032207489 -1
    1 1.072908639907837 1
    2 3.0381953716278076 3
    3 5.003482341766357 5
    4 6.968769073486328 7
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

40. Predict Outputs on Test Inputs
    // Predict outputs for a set of test inputs
    const testInputs = [3, 14, 25, 43, 56, 72];
    const testOutputs = testInputs.map(input =>
    model.predict(tf.tensor2d([input], [1, 1]))
    );
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

41. Sample Prediction Results on
    Test Data
    x y inferenced y calculated
    3 5.003482341766357 5
    14 26.62163734436035 27
    25 48.23978805541992 49
    43 83.61495208740234 85
    56 109.16368103027344 111
    72 140.60826110839844 143
    https://github.com/bradley-holt/tfjs-demos
    

    View Slide

42. Applying
    Deep
    Learning
    Data Science
    Expertise
    Computing
    Resources
    High-Quality
    Training Data
    Model Deployment
    Time
    Model Integration
    Inferencing Code
    And more…
    

    View Slide

43. Model Asset Exchange (MAX)
    Classify
    Generate
    Recognize
    https://developer.ibm.com/exchanges/models/
    

    View Slide

44. Choose deployable model
    Deep Learning asset on MAX
    Deploy
    Swagger specification Inference endpoint Metadata endpoint
    Microservice
    Input pre-processing, model execution, and output post-processing
    Deploy
    model
    Use
    model
    https://developer.ibm.com/exchanges/models/
    

    View Slide

45. Deep
    Learning
    at the
    Edge
    Global Network by fdecomite, on Flickr (CC BY 2.0).
    

    View Slide

46. Device
    Deep Learning
    Model
    Data
    Cloud Results
    Results
    Network
    

    View Slide

47. Device
    Deep Learning
    Model
    Data
    Results
    

    View Slide

48. The
    Cloud
    STOP
    Doggy Driver by Eric Sonstroem, on Flickr (CC BY 2.0).
    

    View Slide

49. Edge Device Inferencing*
    A large model with large input or
    output requires a slow and
    unreliable network API call
    July 13, 2018 / © 2018 IBM Corporation
    Large Model
    *Assumes limited edge device capabilities
    Fast
    Inferencing
    A network API request with small
    input and output can achieve fast,
    but unreliable, inferencing
    A small model run locally can
    achieve fast and reliable inferencing
    with large input and output
    Large
    Input/Output
    

    View Slide

50. Converting a
    Pre-Trained
    Model to
    TensorFlow.js
    Portage by ThreeIfByBike, on Flickr (CC BY-SA 2.0).
    

    View Slide

51. https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c
    

    View Slide

52. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/
    

    View Slide

53. Downloading Model Files
    deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
    http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
    

    View Slide

54. Extracting Model Files
    $ tar xzf deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
    $ ls -lh deeplabv3_mnv2_pascal_trainval
    8.4M frozen_inference_graph.pb
    16M model.ckpt-30000.data-00000-of-00001
    18K 2018 model.ckpt-30000.index
    

    View Slide

55. TensorFlow Model Formats
    Frozen Graphs: Encapsulates model data in a single file
    Checkpoints: Allows for resuming of training
    SavedModel: TensorFlow’s Universal serialization format
    HDF5: Used by Keras to store model data
    

    View Slide

56. Exploring the Model
    https://github.com/lutzroeder/Netron
    

    View Slide

57. Model Input and Output
    

    View Slide

58. Model Input and Output
    

    View Slide

59. Model Input and Output
    

    View Slide

60. Model Input and Output
    

    View Slide

61. Model Input and Output
    

    View Slide

62. Installing the TensorFlow.js Converter
    $ pip install tensorflowjs
    

    View Slide

63. Converting the Model
    $ tensorflowjs_converter \
    --input_format=tf_frozen_model \
    --output_node_names='SemanticPredictions' \
    frozen_inference_graph.pb \
    model
    $ ls -lh model
    4.0M group1-shard1of3
    4.0M group1-shard2of3
    27K group1-shard3of3
    63K tensorflowjs_model.pb
    16K weights_manifest.json
    

    View Slide

64. Image Segmenter Demo App
    https://github.com/vabarbosa/tfjs-sandbox
    

    View Slide

65. Including the TensorFlow.js Library
    
    <br/>
    

    View Slide

66. Loading the Model in TensorFlow.js
    const MODEL_URL = '/model/tensorflowjs_model.pb'
    const WEIGHTS_URL = '/model/weights_manifest.json'
    const model = await tf.loadFrozenModel(MODEL_URL, WEIGHTS_URL)
    

    View Slide

67. Pre-Processing Model Input
    // create a tensor from an image
    const imageTensor = tf.fromPixels(imageElement)
    // insert a dimension into the tensor's shape
    const preprocessedInput = imageTensor.expandDims()
    

    View Slide

68. Running Predictions with the Model
    // generates output prediction
    const prediction = model.predict(preprocessedInput)
    

    View Slide

69. Post-Processing Model Output
    // get image width, height from output
    const imageWidth = prediction.shape[2]
    const imageHeight = prediction.shape[1]
    // create a regular array from the model's output prediction (tensor)
    const segMap = Array.from(prediction.dataSync())
    // create a new array with the corresponding segmentation colors
    const segMapColor = segMap.map(seg => colorMap[seg])
    

    View Slide

70. Image Segmenter Demo App
    https://github.com/vabarbosa/tfjs-sandbox
    

    View Slide

71. brain.js ONNX.js
    https://github.com/BrainJS/brain.js | https://github.com/Microsoft/onnxjs
    Other JavaScript ML Libraries
    

    View Slide

72. What will you build
    with deep learning
    at the edge?
    AZ 89A by Kevin Dooley, on Flickr (CC BY 2.0).
    

    View Slide

73. Resources: Demos
    • Veremin: A Video Theremin Based on PoseNet
    https://veremin.mybluemix.net/
    • Veremin—A Browser-based Video Theremin
    Making music visually using TensorFlow.js, PoseNet, and the Web MIDI & Web Audio APIs
    https://medium.com/ibm-watson-data-lab/veremin-a-browser-based-video-theremin-1548b63200c
    • !" magicat
    https://github.com/CODAIT/magicat
    • An Ode to the Terminal: The (Almost) Perfect Tool
    Bringing Deep Learning to the Terminal with Node and TensorFlow.js
    https://medium.com/ibm-watson-data-lab/an-ode-to-the-terminal-the-almost-perfect-tool-3f4e8435b6b1
    • Deploy a deep learning-powered ‘Magic cropping tool’
    Deploy a deep learning-powered 'Magic cropping tool' using pretrained open source models
    https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
    

    View Slide

74. Resources: Demos (cont’d)
    • Taking Selfies (and More) to the Next Level with Open Source Deep Learning Models
    Everything I’ve Learned About Machine Learning I’ve Learned From Machines.
    https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480
    • TensorFlow.js Demos
    https://github.com/bradley-holt/tfjs-demos
    • Bring Machine Learning to the Browser With TensorFlow.js — Parts I, II & III
    https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c
    https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-ii-7555ed9a999e
    https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3
    • TensorFlow.js Model Playground
    Pre-trained models converted to the TensorFlow.js web friendly format
    https://github.com/vabarbosa/tfjs-sandbox
    

    View Slide

75. Resources: Machine Learning
    • TensorFlow
    https://www.tensorflow.org/
    • Keras
    https://keras.io/
    • PyTorch
    https://pytorch.org/
    • Caffe2
    https://caffe2.ai/
    • ONNX
    https://onnx.ai/
    • Netron
    Visualizer for deep learning and machine learning models
    https://github.com/lutzroeder/Netron
    

    View Slide

76. Resources: Edge Machine Learning
    • TensorFlow.js
    https://js.tensorflow.org/
    • TensorFlow Lite
    https://www.tensorflow.org/lite
    • ONNX.js
    https://github.com/Microsoft/onnxjs
    • brain.js
    ! Neural networks in JavaScript
    https://github.com/BrainJS/brain.js
    • Core ML
    https://developer.apple.com/machine-learning/
    

    View Slide

77. Resources: IBM Developer
    • Center for Open-Source Data & AI Technologies (CODAIT)
    http://codait.org/
    • IBM Developer Model Asset Exchange (MAX)
    https://developer.ibm.com/exchanges/models/
    • Model Asset Exchange (MAX) Code Patterns
    https://ibm.biz/max-developers
    • Image Segmenter
    Identify objects in an image, additionally assigning each pixel of the image to a particular object.
    https://developer.ibm.com/exchanges/models/all/max-image-segmenter/
    • Image Segmenter Model Files
    http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
    

    View Slide

78. Resources: IBM Cloud
    • IBM Cloud
    https://ibm.biz/Bd2A5f
    • IBM Watson Studio
    https://www.ibm.com/cloud/watson-studio
    • IBM Watson Machine Learning
    https://www.ibm.com/cloud/machine-learning
    • IBM Watson OpenScale
    https://www.ibm.com/cloud/watson-openscale
    

    View Slide

79. Thank you
    codait.org
    twitter.com/codait_datalab
    medium.com/ibm-watson-data-lab
    github.com/codait
    developer.ibm.com
    

    View Slide

80. View Slide