Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

With TensorFlow.js developers can embed pre-trained deep learning models directly within a Node.js or web application. Deep learning can be used for many tasks including image or audio classification, facial recognition, object recognition, image caption generation, and natural language processing. TensorFlow.js enables inferencing on deep learning models at the edge, removing the dependency on potentially slow and unreliable network API calls.

Bradley Holt

March 15, 2019

More Decks by Bradley Holt

Other Decks in Programming


  1. Deep Learning at the Edge with TensorFlow.js Bradley Holt Program

    Manager, Developer Advocacy Center for Open-Source Data and AI Technologies @BradleyHolt DOC ID / Month XX, 2018 / © 2018 IBM Corporation
  2. Yes

  3. General AI Metal Skull With Terminator Eye by L.C. Nøttaasen,

    on Flickr <https://flic.kr/p/6xh2Dr> (CC BY-SA 2.0).
  4. Broad AI -[ electrIc b88Gal88 ]- by JD Hancock, on

    Flickr <https://flic.kr/p/6x9t1H> (CC BY 2.0).
  5. Narrow AI Danbo on the Lookout by IQRemix, on Flickr

    <https://flic.kr/p/x5oWjP> (CC BY-SA 2.0).
  6. Install magicat $ npm install -g magicat + [email protected] added

    255 packages from 209 contributors in 11.798s
  7. Scan Directory with magicat $ magicat . Scanning directory '/Users/bradleydholt/tfjs-demos/magicat'...

    The image '23895621638_535be71dee_k.jpg' contains the following segments: background, cat. The image '37038669284_899d7784a9_k.jpg' contains the following segments: background, sheep. The image '37489697170_31d05aa027_k.jpg' contains the following segments: background, sheep. The image '37699459356_24fd526a5e_k.jpg' contains the following segments: background, cat. The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse.
  8. background, cat background, sheep background, sheep background, cat background, horse

    Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  9. Save Image Segment with magicat $ magicat 37699976806_5ce694be36_k.jpg --save horse

    The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse. saved 37699976806_5ce694be36_k-horse.png Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  10. Demo: Object Identification and Image Segmentation with the Magic Cropping

    Tool https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
  11. Labeled Training Data Backpropagation Animal photos by Susanne Nilsson, on

    Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0). Output Errors Cat ❌ Not Cat ❌ Cat ❌ Cat Cat
  12. Input Neural Network Inferencing Output Cat Animal photos by Susanne

    Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  13. Neural Network Inferencing Output Not Cat Input Animal photos by

    Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
  14. Machine Learning Libraries The Leeds Library by Michael D Beckwith,

    on Flickr <https://flic.kr/p/r6ip1G> (CC0 1.0).
  15. Training Data (y = 2x - 1) x y -1

    -3 0 -1 1 1 2 3 3 5 4 7 https://github.com/bradley-holt/tfjs-demos
  16. Prepare the Model for Training // Create a sequential model

    const model = tf.sequential(); model.add(tf.layers.dense({units: 1, inputShape: [1]})); // Prepare the model for training model.compile({loss: 'meanSquaredError', optimizer: 'sgd'}); https://github.com/bradley-holt/tfjs-demos
  17. Train the Model // Create synthetic training data (y =

    2x - 1) const xs = tf.tensor2d([-1, 0, 1, 2, 3, 4], [6, 1]); const ys = tf.tensor2d([-3, -1, 1, 3, 5, 7], [6, 1]); // Train the model await model.fit(xs, ys, {epochs: 250}); https://github.com/bradley-holt/tfjs-demos
  18. Predict Outputs on Training Inputs // Predict outputs for training

    inputs const trainingInputs = [-1, 0, 1, 2, 3, 4]; const trainingOutputs = trainingInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos
  19. Sample Prediction Results on Training Data x y inferenced y

    calculated -1 -2.8576648235321045 -3 0 -0.892378032207489 -1 1 1.072908639907837 1 2 3.0381953716278076 3 3 5.003482341766357 5 4 6.968769073486328 7 https://github.com/bradley-holt/tfjs-demos
  20. Predict Outputs on Test Inputs // Predict outputs for a

    set of test inputs const testInputs = [3, 14, 25, 43, 56, 72]; const testOutputs = testInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos
  21. Sample Prediction Results on Test Data x y inferenced y

    calculated 3 5.003482341766357 5 14 26.62163734436035 27 25 48.23978805541992 49 43 83.61495208740234 85 56 109.16368103027344 111 72 140.60826110839844 143 https://github.com/bradley-holt/tfjs-demos
  22. Applying Deep Learning Data Science Expertise Computing Resources High-Quality Training

    Data Model Deployment Time Model Integration Inferencing Code And more…
  23. Choose deployable model Deep Learning asset on MAX Deploy Swagger

    specification Inference endpoint Metadata endpoint Microservice Input pre-processing, model execution, and output post-processing Deploy model Use model https://developer.ibm.com/exchanges/models/
  24. Deep Learning at the Edge Global Network by fdecomite, on

    Flickr <https://flic.kr/p/a1s6VU> (CC BY 2.0).
  25. The Cloud STOP Doggy Driver by Eric Sonstroem, on Flickr

    <https://flic.kr/p/2a3sGJB> (CC BY 2.0).
  26. Edge Device Inferencing* A large model with large input or

    output requires a slow and unreliable network API call July 13, 2018 / © 2018 IBM Corporation Large Model *Assumes limited edge device capabilities Fast Inferencing A network API request with small input and output can achieve fast, but unreliable, inferencing A small model run locally can achieve fast and reliable inferencing with large input and output Large Input/Output
  27. Converting a Pre-Trained Model to TensorFlow.js Portage by ThreeIfByBike, on

    Flickr <https://flic.kr/p/aMyKY2> (CC BY-SA 2.0).
  28. Extracting Model Files $ tar xzf deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz $ ls -lh

    deeplabv3_mnv2_pascal_trainval 8.4M frozen_inference_graph.pb 16M model.ckpt-30000.data-00000-of-00001 18K 2018 model.ckpt-30000.index
  29. TensorFlow Model Formats Frozen Graphs: Encapsulates model data in a

    single file Checkpoints: Allows for resuming of training SavedModel: TensorFlow’s Universal serialization format HDF5: Used by Keras to store model data
  30. Converting the Model $ tensorflowjs_converter \ --input_format=tf_frozen_model \ --output_node_names='SemanticPredictions' \

    frozen_inference_graph.pb \ model $ ls -lh model 4.0M group1-shard1of3 4.0M group1-shard2of3 27K group1-shard3of3 63K tensorflowjs_model.pb 16K weights_manifest.json
  31. Loading the Model in TensorFlow.js const MODEL_URL = '/model/tensorflowjs_model.pb' const

    WEIGHTS_URL = '/model/weights_manifest.json' const model = await tf.loadFrozenModel(MODEL_URL, WEIGHTS_URL)
  32. Pre-Processing Model Input // create a tensor from an image

    const imageTensor = tf.fromPixels(imageElement) // insert a dimension into the tensor's shape const preprocessedInput = imageTensor.expandDims()
  33. Running Predictions with the Model // generates output prediction const

    prediction = model.predict(preprocessedInput)
  34. Post-Processing Model Output // get image width, height from output

    const imageWidth = prediction.shape[2] const imageHeight = prediction.shape[1] // create a regular array from the model's output prediction (tensor) const segMap = Array.from(prediction.dataSync()) // create a new array with the corresponding segmentation colors const segMapColor = segMap.map(seg => colorMap[seg])
  35. What will you build with deep learning at the edge?

    AZ 89A by Kevin Dooley, on Flickr <https://flic.kr/p/CDLaAT> (CC BY 2.0).
  36. Resources: Demos • Veremin: A Video Theremin Based on PoseNet

    https://veremin.mybluemix.net/ • Veremin—A Browser-based Video Theremin Making music visually using TensorFlow.js, PoseNet, and the Web MIDI & Web Audio APIs https://medium.com/ibm-watson-data-lab/veremin-a-browser-based-video-theremin-1548b63200c • !" magicat https://github.com/CODAIT/magicat • An Ode to the Terminal: The (Almost) Perfect Tool Bringing Deep Learning to the Terminal with Node and TensorFlow.js https://medium.com/ibm-watson-data-lab/an-ode-to-the-terminal-the-almost-perfect-tool-3f4e8435b6b1 • Deploy a deep learning-powered ‘Magic cropping tool’ Deploy a deep learning-powered 'Magic cropping tool' using pretrained open source models https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
  37. Resources: Demos (cont’d) • Taking Selfies (and More) to the

    Next Level with Open Source Deep Learning Models Everything I’ve Learned About Machine Learning I’ve Learned From Machines. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480 • TensorFlow.js Demos https://github.com/bradley-holt/tfjs-demos • Bring Machine Learning to the Browser With TensorFlow.js — Parts I, II & III https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-ii-7555ed9a999e https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3 • TensorFlow.js Model Playground Pre-trained models converted to the TensorFlow.js web friendly format https://github.com/vabarbosa/tfjs-sandbox
  38. Resources: Machine Learning • TensorFlow https://www.tensorflow.org/ • Keras https://keras.io/ •

    PyTorch https://pytorch.org/ • Caffe2 https://caffe2.ai/ • ONNX https://onnx.ai/ • Netron Visualizer for deep learning and machine learning models https://github.com/lutzroeder/Netron
  39. Resources: Edge Machine Learning • TensorFlow.js https://js.tensorflow.org/ • TensorFlow Lite

    https://www.tensorflow.org/lite • ONNX.js https://github.com/Microsoft/onnxjs • brain.js ! Neural networks in JavaScript https://github.com/BrainJS/brain.js • Core ML https://developer.apple.com/machine-learning/
  40. Resources: IBM Developer • Center for Open-Source Data & AI

    Technologies (CODAIT) http://codait.org/ • IBM Developer Model Asset Exchange (MAX) https://developer.ibm.com/exchanges/models/ • Model Asset Exchange (MAX) Code Patterns https://ibm.biz/max-developers • Image Segmenter Identify objects in an image, additionally assigning each pixel of the image to a particular object. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/ • Image Segmenter Model Files http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
  41. Resources: IBM Cloud • IBM Cloud https://ibm.biz/Bd2A5f • IBM Watson

    Studio https://www.ibm.com/cloud/watson-studio • IBM Watson Machine Learning https://www.ibm.com/cloud/machine-learning • IBM Watson OpenScale https://www.ibm.com/cloud/watson-openscale