Deep Learning for Android with TensorFlow 1 @margaretmz

@margaretmz | #ML | #GDE Topics ● Overview of Android ML options ● React Native ● ML Kit ● tf.Keras to TFLite to Android 2

@margaretmz | #ML | #GDE TensorFlow on mobile 3 2015 TF open sourced 2016 TF Mobile 2017 TF Lite developer preview 2018 ML Kit 2019 - ML Kit new features - TF Mobile deprecated - TFLite new announcements!!!

@margaretmz | #ML | #GDE On-device ML for mobile 4 How Who Where Native Android (iOS) apps ● Direct deploy to Android ● With ML Kit Android (or iOS) developers React Native Web developers

5 React Native

@margaretmz | #ML | #GDE React Native Support ● Use TF.js ML directly inside React Native with WebGL acceleration ● Load models from the web, or compile into your application Link to demo video | Link to github 6

7 ML Kit for Android Google’s Machine Learning SDK for Mobile

@margaretmz | #ML | #GDE ML Kit ● Brings Google’s ML expertise to mobile developers in a powerful and easy-to-use package. ● Powered by TensorFlow Lite & hosted on Firebase + 8 Lite

@margaretmz | #ML | #GDE ML Kit - base APIs Here are the base APIs you can use for ML Kit 9 9 Image labelling OCR Face detection Barcode scanning Landmark detection Smart reply Object detection & Tracking Translation (56 languages) AutoML g.co/mlkit

@margaretmz | #ML | #GDE ML Kit - custom models For custom models, ML kits offers - ● Dynamic model downloads ● A/B testing (via Firebase remote Configuration) ● Model compression & conversion (from TensorFlow to TF Lite) 10

@margaretmz | #ML | #GDE ML Kit 11 Convert to Bytebuffer/ bitmap Calibration Java Native Frame Scheduler (Image Timestamp) Convert to byte array Output Results Pipeline config Convert to Grayscale Resize/Rotate Tracker Frame Selection Convert to RGB/Resize/ Rotate Detector (TF Lite model) Object Manager Image Validation Resize Pipeline Classifier ( TF Lite model) Source: ML Kit team

@margaretmz | #ML | #GDE ● Firebase console ● AutoML - train model ● Download TFLite ● Mobile https://firebase.google.com/docs/ml-kit/automl-image-labeling ML Kit - AutoML 12

13 Native Android Apps Direct deploy to Android End to end tf.Keras to TFLite to Android

@margaretmz | #ML | #GDE Models How to get a model for your mobile app: ● Download a pre-trained model (here): Inception-v3, mobilenet etc. ● Transfer learning with a pre-trained model ○ Feature extraction or fine tuning on pre-trained model ○ TensorFlow hub (https://www.tensorflow.org/hub/) ● Train your own model from scratch (example in this talk) 14

15 Training a model With tf.Keras

@margaretmz | #ML | #GDE End to End: tf.Keras to TFLite to Android 16

@margaretmz | #ML | #GDE MNIST dataset ● 60,000 train set and 10,000 test set ● 28x28x1 grayscale images ● 10 classes: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 ● Popular for computer vision ○ “hello world” tutorial or ○ benchmarking ML algorithms 17

@margaretmz | #ML | #GDE Training model Launch sample code on Colab → mnist_tfkeras_to_tflite.ipynb 1. Import data 2. Define a model 3. Train a model 4. Save a Keras model (save as SavedModel) & convert to tflite 18

@margaretmz | #ML | #GDE A simple CNN architecture MNIST example: ● Convolutional layer (definition) ● Pooling layer (definition) ● Dense (fully-connected layer) definition 19 input conv pool conv pool conv pool Dense 0 1 2 3 4 5 6 7 8 9

@margaretmz | #ML | #GDE Inspect the model After defining the model architecture, use model.summary() to show the model architecture 20

@margaretmz | #ML | #GDE Visualize model architecture Use a visualization tool: ● (during training) TensorBoard ● (after conversion) Netron - Drop the .tflite model into Netron and see the model visually Note: model metadata in a new TFLite tool (to be launched) will allow you to inspect the model & modify the metadata 21

@margaretmz | #ML | #GDE TensorFlow 2.0 22 ● tf.Keras (TensorFlow) ● Python libraries: Numpy, Matplotlib etc SavedModel ● Cloud ● Web ● Mobile ● IoT ● Micro controllers ● Edge TPU Training Inference (Or a tf.Keras model)

@margaretmz | #ML | #GDE Model saving When to save as SavedModel or a Keras model? Note: In TensorFlow 2.0 , tf.keras.Model.save() and tf.keras.models.save_model() default to the SavedModel format (not HDF5). (link to doc) 23 SavedModel Keras Model Share pre-trained models and model pieces on TensorFlow Hub Train with tf.Keras and you know your deploy your target When you don’t know the deploy target

24 Conversion With TFLite converter

@margaretmz | #ML | #GDE TensorFlow Lite ● Converter - convert to TFLite file format ● Interpreter - execute inference & optimized for small devices ● Ops/Kernel - limited ops ● Interface to hardware acceleration ○ NN API ○ Edge TPU 25

@margaretmz | #ML | #GDE Changes to TFLite converter 26 TensorFlow 1.X TensorFlow 2.0 from_session() Removed N/A from_frozen_graph() Removed N/A from_saved_model() Same from_saved_model() from_keras_model_file(string) Changed Changed to from_keras_model(model) N/A Added from_concrete_function() Details of changes between TF 1.X and 2.0

@margaretmz | #ML | #GDE TFLite converter 27 Command line Python code (recommended) SavedModel tflite_convert \ --saved_model_dir=/tmp/my_saved_model \ --output_file=/tmp/my_model.tflite Keras Model --keras_model_file=/tmp/my_keras_model.h5 \ --output_file=/tmp/my_model.tflite # Create a converter converter = tf.contrib.lite.TFLiteConverter.from_keras_model_file(keras_model) from_keras_model(model) # Set quantize to true (optional) converter.post_training_quantize=True # Convert the model tflite_model = converter.convert() # Create the tflite model file tflite_model_name = "my_model.tflite" open(tflite_model_name, "wb").write(tflite_model)

@margaretmz | #ML | #GDE Validate TFLite model after conversion 28 Protip: validate the tflite model in python after conversion - 28 TensorFlow result TFLite result Compare results # Test the TensorFlow model on random Input data. tf_result = model(tf.constant(input_data)) # Load TFLite model and allocate tensors. interpreter = tf.lite.Interpreter(model_path="converted_model.tflite") interpreter.allocate_tensors() # Get input and output tensors. input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() # Test model on random input data. input_shape = input_details[0]['shape'] input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32) interpreter.set_tensor(input_details[0]['index'], input_data) interpreter.invoke() tflite_result = interpreter.get_tensor(output_details[0]['index']) # Compare the result. for tf_result, tflite_result in zip(tf_result, tflite_result): np.testing.assert_almost_equal(tf_result, tflite_result, decimal=5)

29 Deploy to Android In Android Studio

@margaretmz | #ML | #GDE TFLite on Android Android sample code DigitRecognizer, step by step: ● Place tf.lite model under assets folder ● Update build.gradle dependencies ● Input image - custom view, gallery or camera ● Data preprocessing ● Classify with the model ● Post processing ● Display result in UI 30

@margaretmz | #ML | #GDE Gradle dependency Update build.gradle to include tensorflow lite android { // Make sure model doesn't get compressed when app is compiled aaptOptions { noCompress "tflite" } } dependencies { …. // Add dependency for TensorFlow Lite compile 'org.tensorflow:tensorflow-lite:[version-number]’ } Place the mnist.tflite model file under /assets folder 31

@margaretmz | #ML | #GDE Input image data Input to the classifier is an image, your options: 1. Draw on canvas from a custom View 2. Get image from Gallery or a 3rd party camera 3. Live frames from Camera2 API Make sure the image dimensions (shape) matches what your classifier expects ● 28x28x1- MNIST or FASHION_MNIST gray scale image ● 299x299x3 - Inception V3 ● 256x256x3 - MobileNet 32

@margaretmz | #ML | #GDE Image preprocessing ● Convert Bitmap to ByteBuffer ● Normalize pixel values to be a certain range ● Convert from color to grayscale, if needed Note: with the TFLite Android Support library announced today, this step is simplified 33

@margaretmz | #ML | #GDE Run inference Load the model file located under the assets folder Use the TensorFlow Lite interpreter to run inference on the input image 34

@margaretmz | #ML | #GDE Post processing & UI The output is an array of probabilities, each correspond to a category Find the category with the highest probability and output result to UI 35

@margaretmz | #ML | #GDE Summary ● Training with tf.Keras is easy ● Model conversion to TFLite is easier ● Android implementation is still challenging & error-prone: (getting improved with the support library) ○ Validate tflite model before deploy to Android ○ Image pre-processing ○ Input tensor shape? ○ Color or grayscale? ○ Post processing My blog post: E2E tf.Keras to TFLite to Android 36

@margaretmz | #ML | #GDE More TFLite examples 37

@margaretmz | #ML | #GDE What’s next? ● Why the future of machine learning is tiny? - Pete Warden ● Deploying to mobile and IoT will get much easier ● TFLite new features ● Federated learning ● On device training 38

@margaretmz | #ML | #GDE Thank you! Follow me on Twitter, Medium or GitHub to learn more about Deep learning, TensorFlow and Android ML. @margaretmz @margaretmz margaretmz 39