Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

With TensorFlow.js developers can embed pre-trained deep learning models directly within a Node.js or web application. Deep learning can be used for many tasks including image or audio classification, facial recognition, object recognition, image caption generation, and natural language processing. TensorFlow.js enables inferencing on deep learning models at the edge, removing the dependency on potentially slow and unreliable network API calls.

8896271ee9d6f46d4b1783c9566cb645?s=128

Bradley Holt

March 15, 2019
Tweet

Transcript

  1. Deep Learning at the Edge with TensorFlow.js Bradley Holt Program

    Manager, Developer Advocacy Center for Open-Source Data and AI Technologies @BradleyHolt DOC ID / Month XX, 2018 / © 2018 IBM Corporation
  2. Is AI overhyped?

  3. Yes

  4. Thank you for coming to my talk

  5. General AI Metal Skull With Terminator Eye by L.C. Nøttaasen,

    on Flickr <https://flic.kr/p/6xh2Dr> (CC BY-SA 2.0).
  6. Broad AI -[ electrIc b88Gal88 ]- by JD Hancock, on

    Flickr <https://flic.kr/p/6x9t1H> (CC BY 2.0).
  7. Narrow AI Danbo on the Lookout by IQRemix, on Flickr

    <https://flic.kr/p/x5oWjP> (CC BY-SA 2.0).
  8. Deep Learning http://codait.org/

  9. Demo: Veremin, A Video Theremin Based on PoseNet https://veremin.mybluemix.net/

  10. None
  11. Try Veremin Yourself https://veremin.mybluemix.net/

  12. Demo: Object Identification and Image Segmentation https://github.com/CODAIT/magicat

  13. https://github.com/CODAIT/magicat

  14. Install magicat $ npm install -g magicat + magicat@1.4.9 added

    255 packages from 209 contributors in 11.798s
  15. 23895621638_535be71dee_k.jpg 37038669284_899d7784a9_k.jpg 37489697170_31d05aa027_k.jpg 37699459356_24fd526a5e_k.jpg 37699976806_5ce694be36_k.jpg Animal photos by Susanne Nilsson,

    on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  16. Scan Directory with magicat $ magicat . Scanning directory '/Users/bradleydholt/tfjs-demos/magicat'...

    The image '23895621638_535be71dee_k.jpg' contains the following segments: background, cat. The image '37038669284_899d7784a9_k.jpg' contains the following segments: background, sheep. The image '37489697170_31d05aa027_k.jpg' contains the following segments: background, sheep. The image '37699459356_24fd526a5e_k.jpg' contains the following segments: background, cat. The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse.
  17. background, cat background, sheep background, sheep background, cat background, horse

    Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  18. Save Image Segment with magicat $ magicat 37699976806_5ce694be36_k.jpg --save horse

    The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse. saved 37699976806_5ce694be36_k-horse.png Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  19. Demo: Object Identification and Image Segmentation with the Magic Cropping

    Tool https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
  20. https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/

  21. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480

  22. Deep Learning Use Cases https://ibm.biz/max-developers

  23. Software Programming Untitled by Marcin Wichary, on Flickr <https://flic.kr/p/68Lhc9> (CC

    BY 2.0).
  24. 0101100100101100000110101101 0101001001100000110101101011 0110010010101010011101001011 0101010011100101010101010000 1010101011100000011010101010 1001000110000011010011010101 0010001011010101011101000101 Software Program Business

    Logic
  25. 0101100100101100000110101101 0101001001100000110101101011 0110010010101010011101001011 0101010011100101010101010000 1010101011100000011010101010 1001000110000011010011010101 0010001011010101011101000101 Software Program Input

    Program Execution Output
  26. Deep Learning

  27. Input Layer Hidden Layers Output Layer

  28. Labeled Training Data Backpropagation Animal photos by Susanne Nilsson, on

    Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0). Output Errors Cat ❌ Not Cat ❌ Cat ❌ Cat Cat
  29. Input Neural Network Inferencing Output Cat Animal photos by Susanne

    Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  30. Neural Network Inferencing Output Not Cat Input Animal photos by

    Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  31. Machine Learning Libraries The Leeds Library by Michael D Beckwith,

    on Flickr <https://flic.kr/p/r6ip1G> (CC0 1.0).
  32. None
  33. None
  34. Demo: Minimal Train and Predict with TensorFlow.js https://github.com/bradley-holt/tfjs-demos

  35. Training Data (y = 2x - 1) x y -1

    -3 0 -1 1 1 2 3 3 5 4 7 https://github.com/bradley-holt/tfjs-demos
  36. Prepare the Model for Training // Create a sequential model

    const model = tf.sequential(); model.add(tf.layers.dense({units: 1, inputShape: [1]})); // Prepare the model for training model.compile({loss: 'meanSquaredError', optimizer: 'sgd'}); https://github.com/bradley-holt/tfjs-demos
  37. Train the Model // Create synthetic training data (y =

    2x - 1) const xs = tf.tensor2d([-1, 0, 1, 2, 3, 4], [6, 1]); const ys = tf.tensor2d([-3, -1, 1, 3, 5, 7], [6, 1]); // Train the model await model.fit(xs, ys, {epochs: 250}); https://github.com/bradley-holt/tfjs-demos
  38. Predict Outputs on Training Inputs // Predict outputs for training

    inputs const trainingInputs = [-1, 0, 1, 2, 3, 4]; const trainingOutputs = trainingInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos
  39. Sample Prediction Results on Training Data x y inferenced y

    calculated -1 -2.8576648235321045 -3 0 -0.892378032207489 -1 1 1.072908639907837 1 2 3.0381953716278076 3 3 5.003482341766357 5 4 6.968769073486328 7 https://github.com/bradley-holt/tfjs-demos
  40. Predict Outputs on Test Inputs // Predict outputs for a

    set of test inputs const testInputs = [3, 14, 25, 43, 56, 72]; const testOutputs = testInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos
  41. Sample Prediction Results on Test Data x y inferenced y

    calculated 3 5.003482341766357 5 14 26.62163734436035 27 25 48.23978805541992 49 43 83.61495208740234 85 56 109.16368103027344 111 72 140.60826110839844 143 https://github.com/bradley-holt/tfjs-demos
  42. Applying Deep Learning Data Science Expertise Computing Resources High-Quality Training

    Data Model Deployment Time Model Integration Inferencing Code And more…
  43. Model Asset Exchange (MAX) Classify Generate Recognize https://developer.ibm.com/exchanges/models/

  44. Choose deployable model Deep Learning asset on MAX Deploy Swagger

    specification Inference endpoint Metadata endpoint Microservice Input pre-processing, model execution, and output post-processing Deploy model Use model https://developer.ibm.com/exchanges/models/
  45. Deep Learning at the Edge Global Network by fdecomite, on

    Flickr <https://flic.kr/p/a1s6VU> (CC BY 2.0).
  46. Device Deep Learning Model Data Cloud Results Results Network

  47. Device Deep Learning Model Data Results

  48. The Cloud STOP Doggy Driver by Eric Sonstroem, on Flickr

    <https://flic.kr/p/2a3sGJB> (CC BY 2.0).
  49. Edge Device Inferencing* A large model with large input or

    output requires a slow and unreliable network API call July 13, 2018 / © 2018 IBM Corporation Large Model *Assumes limited edge device capabilities Fast Inferencing A network API request with small input and output can achieve fast, but unreliable, inferencing A small model run locally can achieve fast and reliable inferencing with large input and output Large Input/Output
  50. Converting a Pre-Trained Model to TensorFlow.js Portage by ThreeIfByBike, on

    Flickr <https://flic.kr/p/aMyKY2> (CC BY-SA 2.0).
  51. https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c

  52. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/

  53. Downloading Model Files deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz

  54. Extracting Model Files $ tar xzf deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz $ ls -lh

    deeplabv3_mnv2_pascal_trainval 8.4M frozen_inference_graph.pb 16M model.ckpt-30000.data-00000-of-00001 18K 2018 model.ckpt-30000.index
  55. TensorFlow Model Formats Frozen Graphs: Encapsulates model data in a

    single file Checkpoints: Allows for resuming of training SavedModel: TensorFlow’s Universal serialization format HDF5: Used by Keras to store model data
  56. Exploring the Model https://github.com/lutzroeder/Netron

  57. Model Input and Output

  58. Model Input and Output

  59. Model Input and Output

  60. Model Input and Output

  61. Model Input and Output

  62. Installing the TensorFlow.js Converter $ pip install tensorflowjs

  63. Converting the Model $ tensorflowjs_converter \ --input_format=tf_frozen_model \ --output_node_names='SemanticPredictions' \

    frozen_inference_graph.pb \ model $ ls -lh model 4.0M group1-shard1of3 4.0M group1-shard2of3 27K group1-shard3of3 63K tensorflowjs_model.pb 16K weights_manifest.json
  64. Image Segmenter Demo App https://github.com/vabarbosa/tfjs-sandbox

  65. Including the TensorFlow.js Library <!-- load TensorFlow.js --> <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs">

    </script>
  66. Loading the Model in TensorFlow.js const MODEL_URL = '/model/tensorflowjs_model.pb' const

    WEIGHTS_URL = '/model/weights_manifest.json' const model = await tf.loadFrozenModel(MODEL_URL, WEIGHTS_URL)
  67. Pre-Processing Model Input // create a tensor from an image

    const imageTensor = tf.fromPixels(imageElement) // insert a dimension into the tensor's shape const preprocessedInput = imageTensor.expandDims()
  68. Running Predictions with the Model // generates output prediction const

    prediction = model.predict(preprocessedInput)
  69. Post-Processing Model Output // get image width, height from output

    const imageWidth = prediction.shape[2] const imageHeight = prediction.shape[1] // create a regular array from the model's output prediction (tensor) const segMap = Array.from(prediction.dataSync()) // create a new array with the corresponding segmentation colors const segMapColor = segMap.map(seg => colorMap[seg])
  70. Image Segmenter Demo App https://github.com/vabarbosa/tfjs-sandbox

  71. brain.js ONNX.js https://github.com/BrainJS/brain.js | https://github.com/Microsoft/onnxjs Other JavaScript ML Libraries

  72. What will you build with deep learning at the edge?

    AZ 89A by Kevin Dooley, on Flickr <https://flic.kr/p/CDLaAT> (CC BY 2.0).
  73. Resources: Demos • Veremin: A Video Theremin Based on PoseNet

    https://veremin.mybluemix.net/ • Veremin—A Browser-based Video Theremin Making music visually using TensorFlow.js, PoseNet, and the Web MIDI & Web Audio APIs https://medium.com/ibm-watson-data-lab/veremin-a-browser-based-video-theremin-1548b63200c • !" magicat https://github.com/CODAIT/magicat • An Ode to the Terminal: The (Almost) Perfect Tool Bringing Deep Learning to the Terminal with Node and TensorFlow.js https://medium.com/ibm-watson-data-lab/an-ode-to-the-terminal-the-almost-perfect-tool-3f4e8435b6b1 • Deploy a deep learning-powered ‘Magic cropping tool’ Deploy a deep learning-powered 'Magic cropping tool' using pretrained open source models https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
  74. Resources: Demos (cont’d) • Taking Selfies (and More) to the

    Next Level with Open Source Deep Learning Models Everything I’ve Learned About Machine Learning I’ve Learned From Machines. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480 • TensorFlow.js Demos https://github.com/bradley-holt/tfjs-demos • Bring Machine Learning to the Browser With TensorFlow.js — Parts I, II & III https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-ii-7555ed9a999e https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3 • TensorFlow.js Model Playground Pre-trained models converted to the TensorFlow.js web friendly format https://github.com/vabarbosa/tfjs-sandbox
  75. Resources: Machine Learning • TensorFlow https://www.tensorflow.org/ • Keras https://keras.io/ •

    PyTorch https://pytorch.org/ • Caffe2 https://caffe2.ai/ • ONNX https://onnx.ai/ • Netron Visualizer for deep learning and machine learning models https://github.com/lutzroeder/Netron
  76. Resources: Edge Machine Learning • TensorFlow.js https://js.tensorflow.org/ • TensorFlow Lite

    https://www.tensorflow.org/lite • ONNX.js https://github.com/Microsoft/onnxjs • brain.js ! Neural networks in JavaScript https://github.com/BrainJS/brain.js • Core ML https://developer.apple.com/machine-learning/
  77. Resources: IBM Developer • Center for Open-Source Data & AI

    Technologies (CODAIT) http://codait.org/ • IBM Developer Model Asset Exchange (MAX) https://developer.ibm.com/exchanges/models/ • Model Asset Exchange (MAX) Code Patterns https://ibm.biz/max-developers • Image Segmenter Identify objects in an image, additionally assigning each pixel of the image to a particular object. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/ • Image Segmenter Model Files http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
  78. Resources: IBM Cloud • IBM Cloud https://ibm.biz/Bd2A5f • IBM Watson

    Studio https://www.ibm.com/cloud/watson-studio • IBM Watson Machine Learning https://www.ibm.com/cloud/machine-learning • IBM Watson OpenScale https://www.ibm.com/cloud/watson-openscale
  79. Thank you codait.org twitter.com/codait_datalab medium.com/ibm-watson-data-lab github.com/codait developer.ibm.com

  80. None