Slide 1

Slide 1 text

@margaretmz TensorFlow Lite Overview E2E tf.Keras to TFLite to Mobile/IoT @TensorFlow Weekly Testing Stand-up Margaret Maynard-Reid, 6/25/2019

Slide 2

Slide 2 text

@margaretmz | #MachineLearning #GDE About me 2 code | speak | write | organize | teach |

Slide 3

Slide 3 text

@margaretmz | #MachineLearning #GDE Topics ● TensorFlow Lite Overview ● On-device ML E2E: ○ train a model from scratch ○ convert to TFLite ○ deploy to mobile and IoT ● Community discussion - TFLite challenges 3

Slide 4

Slide 4 text

Intro 4

Slide 5

Slide 5 text

@margaretmz | #MachineLearning #GDE Examples of computer vision 5 Generative Adversarial Networks (GANs) Generating new images Image classification Is this a cat? Object detection Drawing bounding boxes around the objects Dance Like @I/O Segmentation, pose, GPU on-device Other examples: - Photos enhancement - Style transfer - OCR - Face keypoints

Slide 6

Slide 6 text

@margaretmz | #MachineLearning #GDE Deep Learning - getting started ● Deep learning Frameworks: ○ TensorFlow (>129k stars on Github) ← most popular! ○ PyTorch ○ Caffe (1 & 2) ○ Theano… ● Languages: Python, Swift, Javascript etc. ● IDE - Colab ● Popular neural networks: ○ CNN (Convolutional Neural Networks) ○ RNN (Recurrent Neural Networks) ○ GAN (Generative Adversarial Networks) ○ ... 6

Slide 7

Slide 7 text

@margaretmz | #MachineLearning #GDE TensorFlow 2.0 - model building APIs TensorFlow is a deep learning framework for both research & production Write TensorFlow code in C++, Python, Java, R, Go, SWIFT, JavaScript Deploy to CPU, GPU, TPU, Mobile, Android Things, Raspberry Pi tf.* tf.layers tf.keras Custom Estimator Premade Estimator ← Low level ← Mid level (moving to tf.keras in TF 2.0) ← High level ← Model in a box ← Distributed execution, tf serving 7 TensorFlow 2.0 Beta just got announced!

Slide 8

Slide 8 text

@margaretmz | #MachineLearning #GDE tf.Keras vs Keras No 1:1 mapping between tf.Keras and Keras 8 tf.keras - part of the TensorFlow core APIs import tensorflow as tf # import TensorFlow from tensorflow import keras # import Keras Keras remains an independent open-source project, with backend: ● TensorFlow (Protip: use tf.keras, instead of Keras + TF as backend) ● Theano ● CNTK...

Slide 9

Slide 9 text

@margaretmz | #MachineLearning #GDE tf.Keras model building APIs ● Sequential - the easiest way ● Functional - more flexibility ● Model subclassing - extend a Model class Learn more in Josh Gordon’s blog: What are Symbolic and Imperative APIs in TensorFlow 2.0? 9

Slide 10

Slide 10 text

@margaretmz | #MachineLearning #GDE Blog.tensorflow.org TensorFlow and ML learning resources Tensorflow.org Deep learning with Python by Francois Chollet TensorFlow on Youtube TensorFlow on Twitter #AskTensorFlow #TensorFlowMeets Collection of interactive ML examples (blogpost | website) 10 Interested in learning about TensorFlow 2.0 and try it out? Read My Notes on TensorFlow 2.0 TensorFlow Dev Summit 2019

Slide 11

Slide 11 text

Tools 11

Slide 12

Slide 12 text

@margaretmz | #MachineLearning #GDE Anaconda, TensorFlow & Keras Why use a virtual environment? Ease of upgrade/downgrade of tensorflow ● Download anaconda here ● Create a new virtual environment $ conda create -n [my-env-name] ● Activate the virtual environment you created $ conda activate [my-env-name] ● Install TensorFlow beta $ pip install tensorflow==2.0.0-beta1 My blog post Anaconda, Jupyter Notebook, TensorFlow, Keras 12

Slide 13

Slide 13 text

@margaretmz | #MachineLearning #GDE Google Colab What is Google Colab? ● Jupyter Notebook ○ stored on Google Drive ○ running on Google’s VM in the cloud ● Free GPU and TPU! ● TensorFlow is already installed ● Save and share from your Drive ● Save directly to GitHub 13 Check out these learning resources ● My blog on Colab ● TF team’s blog on Colab ● Laurence’ Video Build a deep neural network in 4 mins with TensorFlow in Colab ● Paige’s video How to take advantage of GPUs & TPUs for your ML project ● Sam’s blog Keras on TPUs in Colab Launch Colab from colab.research.google.com/

Slide 14

Slide 14 text

@margaretmz | #MachineLearning #GDE TensorBoard in Colab TensorBoard Now integrated in Colab! ● Debug ● Monitor ● Visualize Lab - https://www.tensorflow.org/tensorboard/r2/tensorboard_in_notebooks 14

Slide 15

Slide 15 text

Serving models 15

Slide 16

Slide 16 text

@margaretmz | #MachineLearning #GDE End to end: model training → TFLite → Serving 16 ● tf.Keras (TensorFlow) ● Python libraries: Numpy, Matplotlib etc SavedModel ● Cloud ● Web ● Mobile ● IoT ● Micro controllers ● Edge TPU Training Serving

Slide 17

Slide 17 text

TensorFlow for mobile & IoT What are your options? 17

Slide 18

Slide 18 text

@margaretmz | #MachineLearning #GDE TensorFlow for mobile apps 18 2015 TF open sourced 2016 TF mobile 2017 TF Lite developer preview 2018 ML Kit 2019 TF Lite 1.0 TF Mobile deprecated ML Kit improves

Slide 19

Slide 19 text

@margaretmz | #MachineLearning #GDE Android ML with TensorFlow Your options: ● With ML Kit ○ (Out of the box) Base APIs ○ Custom model ● Direct deploy to Android ○ Custom model 19 Custom Models ● Download pre trained models ● Retrain model ● Train your own from scratch ○ data ○ train ○ convert ○ inference Note: you can use AutoML to train but no easy implementation on mobile until recently

Slide 20

Slide 20 text

@margaretmz | #MachineLearning #GDE ML Kit 20 Brings Google’s ML expertise to mobile developers in a powerful and easy-to-use package. Powered by TF Lite, hosted on Firebase For custom models, ML kits offers ● Dynamic model downloads ● A/B testing (via Firebase remote Configuration) ● Model compression & conversion (from TensorFlow to TF Lite) Base APIs: Learn more about ML Kit here Image labelling OCR Face detection Barcode scanning Landmark detection Smart reply (coming soon) Object detection & Tracking Translation (56 languages) AutoML

Slide 21

Slide 21 text

ML Process for mobile & IoT An overview Datasets Train model Convert to TFLite Deploy for inference 21

Slide 22

Slide 22 text

@margaretmz | #MachineLearning #GDE ML process Process: Data -> train -> convert model -> validate TFLite model -> deploy for inference Training vs inference: on-device ML refers to inference only today 22 Training Inference CPU, GPU, Cloud TPU CPU, GPU, (Edge) TPU

Slide 23

Slide 23 text

@margaretmz | #MachineLearning #GDE Data ● Existing datasets ○ Part of the deep learning framework: ■ MNIST, CIFAR10, FASHION_MNIST, IMDB movie reviews etc ○ Open datasets: ■ MNIST, MS-COCO, IMAGENet, CelebA etc ○ Kaggle datasets: https://www.kaggle.com/datasets ○ Google Dataset search tool: https://toolbox.google.com/datasetsearch ○ TF 2.0: TFDS ● Collect your own data 23

Slide 24

Slide 24 text

@margaretmz | #MachineLearning #GDE ML models Your options of getting a model for your mobile app: ● Download a pre-trained model (here): Inception-v3, mobilenet etc. ● Transfer learning with a pre-trained model ○ Feature extraction or fine tuning on pre-trained model ○ TensorFlow hub (https://www.tensorflow.org/hub/) ● Train your own model from scratch (example in this talk) 24

Slide 25

Slide 25 text

@margaretmz | #MachineLearning #GDE Convert, validate & deploy for inference ● Convert the model to tflite format ● Validate the converted model before deploy ● Deploy for inference 25

Slide 26

Slide 26 text

Train a neural network with tf.Keras 26

Slide 27

Slide 27 text

27

Slide 28

Slide 28 text

@margaretmz | #MachineLearning #GDE MNIST dataset ● 60,000 train set and 10,000 test set ● 28x28x1 grayscale images ● 10 classes: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ● Popular for computer vision ○ “hello world” tutorial or ○ benchmarking ML algorithms 28

Slide 29

Slide 29 text

@margaretmz | #MachineLearning #GDE Training the model in Colab Launch my Colab sample code → mnist_tfkeras_to_tflite.ipynb 1. Import data 2. Define a model 3. Train a model 4. Save a Keras model & convert to tflite 5. Validate the TFLite model 29

Slide 30

Slide 30 text

@margaretmz | #MachineLearning #GDE A typical CNN model architecture MNIST example: ● Convolutional layer (definition) ● Pooling layer (definition) ● Dense (fully-connected layer) definition 30 input conv pool conv pool conv pool Dense 0 1 2 3 4 5 6 7 8 9

Slide 31

Slide 31 text

@margaretmz | #MachineLearning #GDE Inspect the model - in python code In python code, after defining the model architecture, use model.summary() to show the model architecture 31

Slide 32

Slide 32 text

@margaretmz | #MachineLearning #GDE Visualize the model Use a visualization tool: ● TensorBoard ● Netron (https://github.com/lutzroeder/Netron) Drop the .tflite model into Netron and see the model visually 32

Slide 33

Slide 33 text

Convert to tflite model 33

Slide 34

Slide 34 text

@margaretmz | #MachineLearning #GDE TensorFlow Lite ● TensorFlow Lite 1.0 ● Works with Inception & MobileNet ● May not support all operations ● Supports ○ Mobile: Android & IOS ○ Android Things ○ Raspberry Pi ○ Edge TPU ○ Microcontroller 34

Slide 35

Slide 35 text

@margaretmz | #MachineLearning #GDE TensorFlow Lite Converter Convert Keras model → a tflite model with the tflite converter There are two options: 1. Command line 2. Python API Note: ● you can convert from SavedModel as well, ● GraphDef and tf.Session are no longer supported in 2.0 for TFLite conversion. Read details on tflite converter on TF documentation here 35

Slide 36

Slide 36 text

@margaretmz | #MachineLearning #GDE Tflite convert through command line To convert a tf.keras model to a tflite model: $ tflite_convert \ $--output_file=mymodel.tflite \ $ --keras_model_file=mymodel.h5 36

Slide 37

Slide 37 text

@margaretmz | #MachineLearning #GDE Tflite convert through Python code Note: converter API is different between TF 1.13, 1.14, 2.0 Alpha & nightly # Create a converter converter = tf.contrib.lite.TFLiteConverter.from_keras_model_file(keras_model) # Set quantize to true converter.post_training_quantize=True # Convert the model tflite_model = converter.convert() # Create the tflite model file tflite_model_name = "mymodel.tflite" open(tflite_model_name, "wb").write(tflite_model) 37

Slide 38

Slide 38 text

@margaretmz | #MachineLearning #GDE Validate the tflite model Protip: validate the converted tflite model in python before deploying it # Load TFLite model and allocate tensors. interpreter = tf.contrib.lite.Interpreter(model_path="converted_model.tflite") interpreter.allocate_tensors() # Get input and output tensors. input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() # Test model on random input data. input_shape = input_details[0]['shape'] input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32) interpreter.set_tensor(input_details[0]['index'], input_data) interpreter.invoke() output_data = interpreter.get_tensor(output_details[0]['index']) print(output_data) 38

Slide 39

Slide 39 text

Run tflite on Android 39

Slide 40

Slide 40 text

@margaretmz | #MachineLearning #GDE Tflite on Android My Android sample code DigitRecognizer, step by step: ● Place tf.lite model under assets folder ● Update build.gradle ● Input an image ● Data preprocessing ● Classify with the model ● Post processing ● Display result in UI 40

Slide 41

Slide 41 text

@margaretmz | #MachineLearning #GDE Dependencies Update build.gradle to include tensorflow lite android { // Make sure model doesn't get compressed when app is compiled aaptOptions { noCompress "tflite" } } dependencies { …. // Add dependency for TensorFlow Lite compile 'org.tensorflow:tensorflow-lite:[version-number]’ } Place the mnist.tflite model file under /assets folder 41

Slide 42

Slide 42 text

@margaretmz | #MachineLearning #GDE Input - image data Input to the classifier is an image, your options: ● Draw on canvas from custom View ● Get image from Gallery or a 3rd party camera ● Live frames from Camera2 API Make sure the image dimensions (shape) matches what your classifier expects ● 28x28x1- MNIST or FASHION_MNIST gray scale image ● 299x299x3 - Inception V3 ● 256x256x3 - MobileNet 42

Slide 43

Slide 43 text

@margaretmz | #MachineLearning #GDE Image preprocessing ● Convert Bitmap to ByteBuffer ● Normalize pixel values to be a certain range ● Convert from color to grayscale, if needed 43

Slide 44

Slide 44 text

@margaretmz | #MachineLearning #GDE Run inference Load the model file located under the assets folder Use the TensorFlow Lite interpreter to run inference on the input image 44

Slide 45

Slide 45 text

@margaretmz | #MachineLearning #GDE Post processing The output is an array of probabilities, each correspond to a category Find the category with the highest probability and output result to UI 45

Slide 46

Slide 46 text

@margaretmz | #MachineLearning #GDE Summary ● Training with tf.Keras is easy ● Model conversion to TFLite is easier ● Android implementation is still challenging & error-prone: (Hopefully this gets improved in the future!) ○ Validate tflite model before deploy to Android ○ Image pre-processing ○ Input tensor shape? ○ Color or grayscale? ○ Post processing 46

Slide 47

Slide 47 text

@margaretmz | #MachineLearning #GDE TFLite demo app Check out Demo app in TensorFlow repo Clone tensorflow project from github git clone https://www.github.com/tensorflow/tensorflow Then open the tflite Android demo from Android Studio /tensorflow/contrib/lite/java/demo Note: TensorFlow Lite moved out contrib as of 10/31/2018 47

Slide 48

Slide 48 text

@margaretmz | #MachineLearning #GDE Inference with GPU TensorFlow Lite Now Faster with Mobile GPUs (Developer Preview) 48

Slide 49

Slide 49 text

@margaretmz | #MachineLearning #GDE More TFLite examples 49

Slide 50

Slide 50 text

@margaretmz | #MachineLearning #GDE TFLite on microcontroller ● Tiny models on tiny computers ● Consumes much less power than CPUs - days on a coin battery ● Tiny RAM and Flash available ● Opens up voice interface to devs More info here - ● Doc - https://www.tensorflow.org/lite/guide/microcontroller ● Code lab - https://g.co/codelabs/sparkfunTF ● Purchase - https://www.sparkfun.com/products/15170 50

Slide 51

Slide 51 text

@margaretmz | #MachineLearning #GDE Coral edge TPU (beta) - hardware for on-device ML acceleration Link to codelab: https://codelabs.developers.google.com/codelabs/edgetpu-classifier/index.html#0 ● Dev board (+ camera module) ● USB Accelerator (+ camera module + Raspberry Pi) Coral Edge TPU 51

Slide 52

Slide 52 text

@margaretmz | #MachineLearning #GDE Coral Edge TPU MobileNet SSD model running on TPU Inference time: < ~20 ms > ~60 fps 52

Slide 53

Slide 53 text

@margaretmz | #MachineLearning #GDE Coral Edge TPU demo MobileNet SSD model running on CPU Inference time > ~390ms ~ 3fps 53

Slide 54

Slide 54 text

@margaretmz | #MachineLearning #GDE Future trends ● Why the future of machine learning is tiny? - Pete Warden ● End to end ML pipeline TFX ● Deploying to mobile and IoT will get much easier ● TFLite will have many more features ● Federated learning ● On device training 54

Slide 55

Slide 55 text

@margaretmz | #MachineLearning #GDE Thank you! 55 Follow me on Twitter, Medium or GitHub to learn more about Deep learning, TensorFlow and on-device ML @margaretmz @margaretmz margaretmz