Deep Learning at the Edge with TensorFlow.js Bradley Holt Program Manager, Developer Advocacy Center for Open-Source Data and AI Technologies @BradleyHolt DOC ID / Month XX, 2018 / © 2018 IBM Corporation

Is AI overhyped?

Yes

Thank you for coming to my talk

General AI Metal Skull With Terminator Eye by L.C. Nøttaasen, on Flickr (CC BY-SA 2.0).

Broad AI -[ electrIc b88Gal88 ]- by JD Hancock, on Flickr (CC BY 2.0).

Narrow AI Danbo on the Lookout by IQRemix, on Flickr (CC BY-SA 2.0).

Deep Learning http://codait.org/

Demo: Veremin, A Video Theremin Based on PoseNet https://veremin.mybluemix.net/

No content

Try Veremin Yourself https://veremin.mybluemix.net/

Demo: Object Identification and Image Segmentation https://github.com/CODAIT/magicat

https://github.com/CODAIT/magicat

Install magicat $ npm install -g magicat + [email protected] added 255 packages from 209 contributors in 11.798s

23895621638_535be71dee_k.jpg 37038669284_899d7784a9_k.jpg 37489697170_31d05aa027_k.jpg 37699459356_24fd526a5e_k.jpg 37699976806_5ce694be36_k.jpg Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

Scan Directory with magicat $ magicat . Scanning directory '/Users/bradleydholt/tfjs-demos/magicat'... The image '23895621638_535be71dee_k.jpg' contains the following segments: background, cat. The image '37038669284_899d7784a9_k.jpg' contains the following segments: background, sheep. The image '37489697170_31d05aa027_k.jpg' contains the following segments: background, sheep. The image '37699459356_24fd526a5e_k.jpg' contains the following segments: background, cat. The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse.

background, cat background, sheep background, sheep background, cat background, horse Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

Save Image Segment with magicat $ magicat 37699976806_5ce694be36_k.jpg --save horse The image '37699976806_5ce694be36_k.jpg' contains the following segments: background, horse. saved 37699976806_5ce694be36_k-horse.png Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

Demo: Object Identification and Image Segmentation with the Magic Cropping Tool https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/

https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/

https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480

Deep Learning Use Cases https://ibm.biz/max-developers

Software Programming Untitled by Marcin Wichary, on Flickr (CC BY 2.0).

0101100100101100000110101101 0101001001100000110101101011 0110010010101010011101001011 0101010011100101010101010000 1010101011100000011010101010 1001000110000011010011010101 0010001011010101011101000101 Software Program Business Logic

0101100100101100000110101101 0101001001100000110101101011 0110010010101010011101001011 0101010011100101010101010000 1010101011100000011010101010 1001000110000011010011010101 0010001011010101011101000101 Software Program Input Program Execution Output

Deep Learning

Input Layer Hidden Layers Output Layer

Labeled Training Data Backpropagation Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0). Output Errors Cat ❌ Not Cat ❌ Cat ❌ Cat Cat

Input Neural Network Inferencing Output Cat Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

Neural Network Inferencing Output Not Cat Input Animal photos by Susanne Nilsson, on Flickr (CC BY-SA 2.0).

Machine Learning Libraries The Leeds Library by Michael D Beckwith, on Flickr (CC0 1.0).

No content

No content

Demo: Minimal Train and Predict with TensorFlow.js https://github.com/bradley-holt/tfjs-demos

Training Data (y = 2x - 1) x y -1 -3 0 -1 1 1 2 3 3 5 4 7 https://github.com/bradley-holt/tfjs-demos

Prepare the Model for Training // Create a sequential model const model = tf.sequential(); model.add(tf.layers.dense({units: 1, inputShape: [1]})); // Prepare the model for training model.compile({loss: 'meanSquaredError', optimizer: 'sgd'}); https://github.com/bradley-holt/tfjs-demos

Train the Model // Create synthetic training data (y = 2x - 1) const xs = tf.tensor2d([-1, 0, 1, 2, 3, 4], [6, 1]); const ys = tf.tensor2d([-3, -1, 1, 3, 5, 7], [6, 1]); // Train the model await model.fit(xs, ys, {epochs: 250}); https://github.com/bradley-holt/tfjs-demos

Predict Outputs on Training Inputs // Predict outputs for training inputs const trainingInputs = [-1, 0, 1, 2, 3, 4]; const trainingOutputs = trainingInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos

Sample Prediction Results on Training Data x y inferenced y calculated -1 -2.8576648235321045 -3 0 -0.892378032207489 -1 1 1.072908639907837 1 2 3.0381953716278076 3 3 5.003482341766357 5 4 6.968769073486328 7 https://github.com/bradley-holt/tfjs-demos

Predict Outputs on Test Inputs // Predict outputs for a set of test inputs const testInputs = [3, 14, 25, 43, 56, 72]; const testOutputs = testInputs.map(input => model.predict(tf.tensor2d([input], [1, 1])) ); https://github.com/bradley-holt/tfjs-demos

Sample Prediction Results on Test Data x y inferenced y calculated 3 5.003482341766357 5 14 26.62163734436035 27 25 48.23978805541992 49 43 83.61495208740234 85 56 109.16368103027344 111 72 140.60826110839844 143 https://github.com/bradley-holt/tfjs-demos

Applying Deep Learning Data Science Expertise Computing Resources High-Quality Training Data Model Deployment Time Model Integration Inferencing Code And more…

Model Asset Exchange (MAX) Classify Generate Recognize https://developer.ibm.com/exchanges/models/

Choose deployable model Deep Learning asset on MAX Deploy Swagger specification Inference endpoint Metadata endpoint Microservice Input pre-processing, model execution, and output post-processing Deploy model Use model https://developer.ibm.com/exchanges/models/

Deep Learning at the Edge Global Network by fdecomite, on Flickr (CC BY 2.0).

Device Deep Learning Model Data Cloud Results Results Network

Device Deep Learning Model Data Results

The Cloud STOP Doggy Driver by Eric Sonstroem, on Flickr (CC BY 2.0).

Edge Device Inferencing* A large model with large input or output requires a slow and unreliable network API call July 13, 2018 / © 2018 IBM Corporation Large Model *Assumes limited edge device capabilities Fast Inferencing A network API request with small input and output can achieve fast, but unreliable, inferencing A small model run locally can achieve fast and reliable inferencing with large input and output Large Input/Output

Converting a Pre-Trained Model to TensorFlow.js Portage by ThreeIfByBike, on Flickr (CC BY-SA 2.0).

https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c

https://developer.ibm.com/exchanges/models/all/max-image-segmenter/

Downloading Model Files deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz

Extracting Model Files $ tar xzf deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz $ ls -lh deeplabv3_mnv2_pascal_trainval 8.4M frozen_inference_graph.pb 16M model.ckpt-30000.data-00000-of-00001 18K 2018 model.ckpt-30000.index

TensorFlow Model Formats Frozen Graphs: Encapsulates model data in a single file Checkpoints: Allows for resuming of training SavedModel: TensorFlow’s Universal serialization format HDF5: Used by Keras to store model data

Exploring the Model https://github.com/lutzroeder/Netron

Model Input and Output

Model Input and Output

Model Input and Output

Model Input and Output

Model Input and Output

Installing the TensorFlow.js Converter $ pip install tensorflowjs

Converting the Model $ tensorflowjs_converter \ --input_format=tf_frozen_model \ --output_node_names='SemanticPredictions' \ frozen_inference_graph.pb \ model $ ls -lh model 4.0M group1-shard1of3 4.0M group1-shard2of3 27K group1-shard3of3 63K tensorflowjs_model.pb 16K weights_manifest.json

Image Segmenter Demo App https://github.com/vabarbosa/tfjs-sandbox

Including the TensorFlow.js Library

Loading the Model in TensorFlow.js const MODEL_URL = '/model/tensorflowjs_model.pb' const WEIGHTS_URL = '/model/weights_manifest.json' const model = await tf.loadFrozenModel(MODEL_URL, WEIGHTS_URL)

Pre-Processing Model Input // create a tensor from an image const imageTensor = tf.fromPixels(imageElement) // insert a dimension into the tensor's shape const preprocessedInput = imageTensor.expandDims()

Running Predictions with the Model // generates output prediction const prediction = model.predict(preprocessedInput)

Post-Processing Model Output // get image width, height from output const imageWidth = prediction.shape[2] const imageHeight = prediction.shape[1] // create a regular array from the model's output prediction (tensor) const segMap = Array.from(prediction.dataSync()) // create a new array with the corresponding segmentation colors const segMapColor = segMap.map(seg => colorMap[seg])

Image Segmenter Demo App https://github.com/vabarbosa/tfjs-sandbox

brain.js ONNX.js https://github.com/BrainJS/brain.js | https://github.com/Microsoft/onnxjs Other JavaScript ML Libraries

What will you build with deep learning at the edge? AZ 89A by Kevin Dooley, on Flickr (CC BY 2.0).

Resources: Demos • Veremin: A Video Theremin Based on PoseNet https://veremin.mybluemix.net/ • Veremin—A Browser-based Video Theremin Making music visually using TensorFlow.js, PoseNet, and the Web MIDI & Web Audio APIs https://medium.com/ibm-watson-data-lab/veremin-a-browser-based-video-theremin-1548b63200c • !" magicat https://github.com/CODAIT/magicat • An Ode to the Terminal: The (Almost) Perfect Tool Bringing Deep Learning to the Terminal with Node and TensorFlow.js https://medium.com/ibm-watson-data-lab/an-ode-to-the-terminal-the-almost-perfect-tool-3f4e8435b6b1 • Deploy a deep learning-powered ‘Magic cropping tool’ Deploy a deep learning-powered 'Magic cropping tool' using pretrained open source models https://developer.ibm.com/patterns/max-image-segmenter-magic-cropping-tool-web-app/

Resources: Demos (cont’d) • Taking Selfies (and More) to the Next Level with Open Source Deep Learning Models Everything I’ve Learned About Machine Learning I’ve Learned From Machines. https://medium.com/ibm-watson-data-lab/taking-selfies-and-more-to-the-next-level-with-open-source-deep-learning-models-de9ac8da4480 • TensorFlow.js Demos https://github.com/bradley-holt/tfjs-demos • Bring Machine Learning to the Browser With TensorFlow.js — Parts I, II & III https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-i-16924457291c https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-ii-7555ed9a999e https://medium.com/ibm-watson-data-lab/bring-machine-learning-to-the-browser-with-tensorflow-js-part-iii-62d2b09b10a3 • TensorFlow.js Model Playground Pre-trained models converted to the TensorFlow.js web friendly format https://github.com/vabarbosa/tfjs-sandbox

Resources: Machine Learning • TensorFlow https://www.tensorflow.org/ • Keras https://keras.io/ • PyTorch https://pytorch.org/ • Caffe2 https://caffe2.ai/ • ONNX https://onnx.ai/ • Netron Visualizer for deep learning and machine learning models https://github.com/lutzroeder/Netron

Resources: Edge Machine Learning • TensorFlow.js https://js.tensorflow.org/ • TensorFlow Lite https://www.tensorflow.org/lite • ONNX.js https://github.com/Microsoft/onnxjs • brain.js ! Neural networks in JavaScript https://github.com/BrainJS/brain.js • Core ML https://developer.apple.com/machine-learning/

Resources: IBM Developer • Center for Open-Source Data & AI Technologies (CODAIT) http://codait.org/ • IBM Developer Model Asset Exchange (MAX) https://developer.ibm.com/exchanges/models/ • Model Asset Exchange (MAX) Code Patterns https://ibm.biz/max-developers • Image Segmenter Identify objects in an image, additionally assigning each pixel of the image to a particular object. https://developer.ibm.com/exchanges/models/all/max-image-segmenter/ • Image Segmenter Model Files http://max-assets.s3-api.us-geo.objectstorage.softlayer.net/deeplab/deeplabv3_mnv2_pascal_trainval_2018_01_29.tar.gz

Resources: IBM Cloud • IBM Cloud https://ibm.biz/Bd2A5f • IBM Watson Studio https://www.ibm.com/cloud/watson-studio • IBM Watson Machine Learning https://www.ibm.com/cloud/machine-learning • IBM Watson OpenScale https://www.ibm.com/cloud/watson-openscale

Thank you codait.org twitter.com/codait_datalab medium.com/ibm-watson-data-lab github.com/codait developer.ibm.com

No content