Deep Learning at the Edge with TensorFlow.js at ConFoo Montreal

With TensorFlow.js developers can embed pre-trained deep learning models directly within a Node.js or web application. Deep learning can be used for many tasks including image or audio classification, facial recognition, object recognition, image caption generation, and natural language processing. TensorFlow.js enables inferencing on deep learning models at the edge, removing the dependency on potentially slow and unreliable network API calls.

Transcript

1. Deep Learning at the Edge with TensorFlow.js Bradley Holt Program Manager, Developer Advocacy Center for Open-Source Data and AI Technologies @BradleyHolt DOC ID / Month XX, 2018 / © 2018 IBM Corporation
2. Is AI overhyped?
3. Yes
4. Thank you for coming to my talk
5. General AI Metal Skull With Terminator Eye by L.C. Nøttaasen, on Flickr <https://flic.kr/p/6xh2Dr> (CC BY-SA 2.0).
6. Broad AI -[ electrIc b88Gal88 ]- by JD Hancock, on Flickr <https://flic.kr/p/6x9t1H> (CC BY 2.0).
7. Narrow AI Danbo on the Lookout by IQRemix, on Flickr <https://flic.kr/p/x5oWjP> (CC BY-SA 2.0).
8. Deep Learning http://codait.org/
9. Demo: Veremin, A Video Theremin Based on PoseNet https://veremin.mybluemix.net/
11. Try Veremin Yourself https://veremin.mybluemix.net/
12. Demo: Object Identification and Image Segmentation https://github.com/CODAIT/magicat
13. https://github.com/CODAIT/magicat
14. Install magicat $ npm install -g magicat + magicat@1.4.9 added 255 packages from 209 contributors in 11.798s
15. 23895621638_535be71dee_k.jpg 37038669284_899d7784a9_k.jpg 37489697170_31d05aa027_k.jpg 37699459356_24fd526a5e_k.jpg 37699976806_5ce694be36_k.jpg Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
16. Scan Directory with magicat $ magicat . Scanning directory '/Users/bradleydholt/tfjs-demos/magicat'... The image '23895621638_535be71dee_k.jpg' contains the following segments: background, cat. The image '37038669284_899d7784a9_k.jpg' contains the following segments: background, sheep. The image '37489697170_31d05aa027_k.jpg' contains the following segments: background, sheep. The image '37699459356_24fd526a5e_k.jpg' contains the following segments: background, cat. The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse.
17. background, cat background, sheep background, sheep background, cat background, horse Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
18. Save Image Segment with magicat $ magicat 37699976806_5ce694be36_k.jpg --save horse The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse. saved 37699976806_5ce694be36_k-horse.png Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
19. Demo: Object Identification and Image Segmentation with the Magic Cropping Tool https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
20. https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
21. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480
22. Deep Learning Use Cases https://ibm.biz/max-developers
23. Software Programming Untitled by Marcin Wichary, on Flickr <https://flic.kr/p/68Lhc9> (CC BY 2.0).
24. 0101100100101100000110101101 0101001001100000110101101011 0110010010101010011101001011 0101010011100101010101010000 1010101011100000011010101010 1001000110000011010011010101 0010001011010101011101000101 Software Program Business Logic
25. 0101100100101100000110101101 0101001001100000110101101011 0110010010101010011101001011 0101010011100101010101010000 1010101011100000011010101010 1001000110000011010011010101 0010001011010101011101000101 Software Program Input Program Execution Output
26. Deep Learning
27. Input Layer Hidden Layers Output Layer
28. Labeled Training Data Backpropagation Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0). Output Errors Cat ❌ Not Cat ❌ Cat ❌ Cat Cat
29. Input Neural Network Inferencing Output Cat Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
30. Neural Network Inferencing Output Not Cat Input Animal photos by Susanne Nilsson, on Flickr <https://www.flickr.com/photos/infomastern/> (CC BY-SA 2.0).
31. Machine Learning Libraries The Leeds Library by Michael D Beckwith, on Flickr <https://flic.kr/p/r6ip1G> (CC0 1.0).
34. Demo: Minimal Train and Predict with TensorFlow.js https://github.com/bradley-holt/tfjs-demos
35. Training Data (y = 2x - 1) x y -1 -3 0 -1 1 1 2 3 3 5 4 7 https://github.com/bradley-holt/tfjs-demos
36. Prepare the Model for Training // Create a sequential model const model = tf.sequential(); model.add(tf.layers.dense({units: 1, inputShape: [1]})); // Prepare the model for training model.compile({loss: 'meanSquaredError', optimizer: 'sgd'}); https://github.com/bradley-holt/tfjs-demos
37. Train the Model // Create synthetic training data (y = 2x - 1) const xs = tf.tensor2d([-1, 0, 1, 2, 3, 4], [6, 1]); const ys = tf.tensor2d([-3, -1, 1, 3, 5, 7], [6, 1]); // Train the model await model.fit(xs, ys, {epochs: 250}); https://github.com/bradley-holt/tfjs-demos
38. Predict Outputs on Training Inputs // Predict outputs for training inputs const trainingInputs = [-1, 0, 1, 2, 3, 4]; const trainingOutputs = trainingInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos
39. Sample Prediction Results on Training Data x y inferenced y calculated -1 -2.8576648235321045 -3 0 -0.892378032207489 -1 1 1.072908639907837 1 2 3.0381953716278076 3 3 5.003482341766357 5 4 6.968769073486328 7 https://github.com/bradley-holt/tfjs-demos
40. Predict Outputs on Test Inputs // Predict outputs for a set of test inputs const testInputs = [3, 14, 25, 43, 56, 72]; const testOutputs = testInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos
41. Sample Prediction Results on Test Data x y inferenced y calculated 3 5.003482341766357 5 14 26.62163734436035 27 25 48.23978805541992 49 43 83.61495208740234 85 56 109.16368103027344 111 72 140.60826110839844 143 https://github.com/bradley-holt/tfjs-demos
42. Applying Deep Learning Data Science Expertise Computing Resources High-Quality Training Data Model Deployment Time Model Integration Inferencing Code And more…
43. Model Asset Exchange (MAX) Classify Generate Recognize https://developer.ibm.com/exchanges/models/
44. Choose deployable model Deep Learning asset on MAX Deploy Swagger specification Inference endpoint Metadata endpoint Microservice Input pre-processing, model execution, and output post-processing Deploy model Use model https://developer.ibm.com/exchanges/models/
45. Deep Learning at the Edge Global Network by fdecomite, on Flickr <https://flic.kr/p/a1s6VU> (CC BY 2.0).
46. Device Deep Learning Model Data Cloud Results Results Network
47. Device Deep Learning Model Data Results
48. The Cloud STOP Doggy Driver by Eric Sonstroem, on Flickr <https://flic.kr/p/2a3sGJB> (CC BY 2.0).
49. Edge Device Inferencing* A large model with large input or output requires a slow and unreliable network API call July 13, 2018 / © 2018 IBM Corporation Large Model *Assumes limited edge device capabilities Fast Inferencing A network API request with small input and output can achieve fast, but unreliable, inferencing A small model run locally can achieve fast and reliable inferencing with large input and output Large Input/Output
50. Converting a Pre-Trained Model to TensorFlow.js Portage by ThreeIfByBike, on Flickr <https://flic.kr/p/aMyKY2> (CC BY-SA 2.0).
51. https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c
52. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/
53. Downloading Model Files deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
54. Extracting Model Files $ tar xzf deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz $ ls -lh deeplabv3_mnv2_pascal_trainval 8.4M frozen_inference_graph.pb 16M model.ckpt-30000.data-00000-of-00001 18K 2018 model.ckpt-30000.index
55. TensorFlow Model Formats Frozen Graphs: Encapsulates model data in a single file Checkpoints: Allows for resuming of training SavedModel: TensorFlow’s Universal serialization format HDF5: Used by Keras to store model data
56. Exploring the Model https://github.com/lutzroeder/Netron
57. Model Input and Output
58. Model Input and Output
59. Model Input and Output
60. Model Input and Output
61. Model Input and Output
62. Installing the TensorFlow.js Converter $ pip install tensorflowjs
63. Converting the Model $ tensorflowjs_converter \ --input_format=tf_frozen_model \ --output_node_names='SemanticPredictions' \ frozen_inference_graph.pb \ model $ ls -lh model 4.0M group1-shard1of3 4.0M group1-shard2of3 27K group1-shard3of3 63K tensorflowjs_model.pb 16K weights_manifest.json
64. Image Segmenter Demo App https://github.com/vabarbosa/tfjs-sandbox
65. Including the TensorFlow.js Library <!-- load TensorFlow.js --> <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"> </script>
66. Loading the Model in TensorFlow.js const MODEL_URL = '/model/tensorflowjs_model.pb' const WEIGHTS_URL = '/model/weights_manifest.json' const model = await tf.loadFrozenModel(MODEL_URL, WEIGHTS_URL)
67. Pre-Processing Model Input // create a tensor from an image const imageTensor = tf.fromPixels(imageElement) // insert a dimension into the tensor's shape const preprocessedInput = imageTensor.expandDims()
68. Running Predictions with the Model // generates output prediction const prediction = model.predict(preprocessedInput)
69. Post-Processing Model Output // get image width, height from output const imageWidth = prediction.shape[2] const imageHeight = prediction.shape[1] // create a regular array from the model's output prediction (tensor) const segMap = Array.from(prediction.dataSync()) // create a new array with the corresponding segmentation colors const segMapColor = segMap.map(seg => colorMap[seg])
70. Image Segmenter Demo App https://github.com/vabarbosa/tfjs-sandbox
71. brain.js ONNX.js https://github.com/BrainJS/brain.js | https://github.com/Microsoft/onnxjs Other JavaScript ML Libraries
72. What will you build with deep learning at the edge? AZ 89A by Kevin Dooley, on Flickr <https://flic.kr/p/CDLaAT> (CC BY 2.0).
73. Resources: Demos • Veremin: A Video Theremin Based on PoseNet https://veremin.mybluemix.net/ • Veremin—A Browser-based Video Theremin Making music visually using TensorFlow.js, PoseNet, and the Web MIDI & Web Audio APIs https://medium.com/ibm-watson-data-lab/veremin-a-browser-based-video-theremin-1548b63200c • !" magicat https://github.com/CODAIT/magicat • An Ode to the Terminal: The (Almost) Perfect Tool Bringing Deep Learning to the Terminal with Node and TensorFlow.js https://medium.com/ibm-watson-data-lab/an-ode-to-the-terminal-the-almost-perfect-tool-3f4e8435b6b1 • Deploy a deep learning-powered ‘Magic cropping tool’ Deploy a deep learning-powered 'Magic cropping tool' using pretrained open source models https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/
74. Resources: Demos (cont’d) • Taking Selfies (and More) to the Next Level with Open Source Deep Learning Models Everything I’ve Learned About Machine Learning I’ve Learned From Machines. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480 • TensorFlow.js Demos https://github.com/bradley-holt/tfjs-demos • Bring Machine Learning to the Browser With TensorFlow.js — Parts I, II & III https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-ii-7555ed9a999e https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3 • TensorFlow.js Model Playground Pre-trained models converted to the TensorFlow.js web friendly format https://github.com/vabarbosa/tfjs-sandbox
75. Resources: Machine Learning • TensorFlow https://www.tensorflow.org/ • Keras https://keras.io/ • PyTorch https://pytorch.org/ • Caffe2 https://caffe2.ai/ • ONNX https://onnx.ai/ • Netron Visualizer for deep learning and machine learning models https://github.com/lutzroeder/Netron
76. Resources: Edge Machine Learning • TensorFlow.js https://js.tensorflow.org/ • TensorFlow Lite https://www.tensorflow.org/lite • ONNX.js https://github.com/Microsoft/onnxjs • brain.js ! Neural networks in JavaScript https://github.com/BrainJS/brain.js • Core ML https://developer.apple.com/machine-learning/
77. Resources: IBM Developer • Center for Open-Source Data & AI Technologies (CODAIT) http://codait.org/ • IBM Developer Model Asset Exchange (MAX) https://developer.ibm.com/exchanges/models/ • Model Asset Exchange (MAX) Code Patterns https://ibm.biz/max-developers • Image Segmenter Identify objects in an image, additionally assigning each pixel of the image to a particular object. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/ • Image Segmenter Model Files http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz
78. Resources: IBM Cloud • IBM Cloud https://ibm.biz/Bd2A5f • IBM Watson Studio https://www.ibm.com/cloud/watson-studio • IBM Watson Machine Learning https://www.ibm.com/cloud/machine-learning • IBM Watson OpenScale https://www.ibm.com/cloud/watson-openscale
79. Thank you codait.org twitter.com/codait_datalab medium.com/ibm-watson-data-lab github.com/codait developer.ibm.com