Machine Learning for Mobile with TensorFlow

I gave a talk for GDG Boulder:
https://www.meetup.com/Google-Developer-Group-Boulder/events/257013855/

This talk shares how to make intelligent apps with TensorFlow. I gave an overview of the various options from using ready made APIs to training your own custom models. I walked through the end to end process of how to train a model with TensorFlow high level API tf.Keras, convert to a TensorFlow Lite model and deploy it to an Android app.

Transcript

1. @margaretmz Deep Learning for Android With TensorFlow DevFest Silicon Valley Margaret Maynard-Reid, 2/23/2019
2. @margaretmz | #MachineLearning #GDE | #AndroidDev Slides Slides for this talk is on speakerdeck bit.ly/dl-for-android-with-tf-devfestsv19 Click on download PDF to access the links in downloaded file 2
3. @margaretmz | #MachineLearning #GDE | #AndroidDev Topics • Intro ◦ TensorFlow, tf.Keras, tools • ML on Android, your options ◦ Hosted with ML Kit ◦ Direct deploy to Android app • ML process for mobile: data -> train -> convert -> deploy for inference • End to end tf.Keras to TF Lite to Android ◦ Train a model from scratch with tf.Keras ◦ Convert Keras model to tflite ◦ Inspect & test the tflite model ◦ Run tflite on Android 3
4. Intro AI, ML, Deep Learning Computer Vision TensorFlow, Keras 4
5. @margaretmz | #MachineLearning #GDE | #AndroidDev AI vs. ML vs. Deep Learning Artificial Intelligence Machine Learning Deep Learning: 5 5 - Computer Vision - NLP ….
6. @margaretmz | #MachineLearning #GDE | #AndroidDev Some examples of deep learning NSynth - make music with deep learning 6 Computer Vision • Image classification - is this a cat? • Object detection - self driving car • Generating new images Generating new images using generative adversarial networks (GANs) Is this a cat?
7. @margaretmz | #MachineLearning #GDE | #AndroidDev Get started with Deep Learning • Deep learning Frameworks: ◦ TensorFlow (>100k stars on Github) ← most popular! ◦ PyTorch ◦ Caffe (1 & 2) ◦ Theano… • Languages: Python, Swift, Javascript etc. • IDE - Colab or PyCharm (for Python code) • Type of neural networks: ◦ CNN (Convolutional Neural Networks) ◦ RNN (Recurrent Neural Networks) ◦ GAN (Generative Adversarial Networks) ◦ ... 7
8. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow model building APIs TensorFlow is a deep learning framework for both research & production Write TensorFlow code in C++, Python, Java, R, Go, SWIFT, JavaScript Deploy to CPU, GPU, TPU, Mobile, Android Things, Raspberry Pi tf.* tf.layers tf.keras Custom Estimator Premade Estimator ← Low level ← Mid level (moving to tf.keras in TF 2.0) ← High level ← Model in a box ← Distributed execution, tf serving 8
9. @margaretmz | #MachineLearning #GDE | #AndroidDev tf.Keras vs Keras tf.keras - part of the TensorFlow core APIs import tensorflow as tf keras = tf.keras... Keras remains an independent open-source project, with backend: • TensorFlow (Note: you should just use tf.keras, instead of Keras + TF as backend) • Theano • CNTK... 9
10. @margaretmz | #MachineLearning #GDE | #AndroidDev tf.Keras model building APIs • Sequential - the easiest way • Functional - more flexibility • Model subclassing - extend a Model class Learn more in Josh Gordon’s blog: What are Symbolic and Imperative APIs in TensorFlow 2.0? 10
11. @margaretmz | #MachineLearning #GDE | #AndroidDev Blog.tensorflow.org TensorFlow learning resources Tensorflow.org Deep learning with Python by Francois Chollet TensorFlow on Youtube TensorFlow on Twitter #AskTensorFlow #TensorFlowMeets Collection of interactive ML examples (blogpost | website) 11 Interested in learning about TensorFlow 2.0 preview and try it out? Read My Notes on TensorFlow 2.0 TensorFlow Dev Summit
12. Tools 12
13. @margaretmz | #MachineLearning #GDE | #AndroidDev Tools for deep learning • Anaconda or Virtualenv - for creating virtual environments • Jupyter Notebook • Google Colab • PyCharm 13
14. @margaretmz | #MachineLearning #GDE | #AndroidDev Anaconda, TensorFlow & Keras Why use a virtual environment? Ease of upgrade/downgrade of tensorflow • Download anaconda here • Create a new virtual environment $ conda create -n [my-env-name] • Activate the virtual environment you created $ conda activate [my-env-name] • Install TensorFlow which also contains tf.keras $ pip install --upgrade tensorflow • Launch Jupyter Notebook server $ jupyter notebook My blog post Anaconda, Jupyter Notebook, TensorFlow, Keras 14
15. @margaretmz | #MachineLearning #GDE | #AndroidDev Jupyter Notebook • Interactive programming in the web browser • Great for visualization • Great for collaboration • Popular tool for studying machine learning / deep learning 15
16. @margaretmz | #MachineLearning #GDE | #AndroidDev Google Colab What is Google Colab? • Jupyter Notebook ◦ stored on Google Drive ◦ running on Google’s VM in the cloud • Free GPU and TPU! • TensorFlow is already installed • Save and share from your Drive • Save directly to GitHub 16 Check out these learning resources • My blog on Colab • TF team’s blog on Colab • Laurence’ Video Build a deep neural network in 4 mins with TensorFlow in Colab • Paige’s video How to take advantage of GPUs & TPUs for your ML project • Sam’s blog Keras on TPUs in Colab Launch Colab from colab.research.google.com/
17. @margaretmz | #MachineLearning #GDE | #AndroidDev PyCharm • Made by JetBrains • Similar UI as Android Studio • Run Python scripts in PyCharm ◦ Enable eager execution mode (default in TensorFlow 2.0) ◦ Set breakpoints ◦ Debug! 17
18. Android with TensorFlow What are the options? 18
19. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow for mobile apps History of TensorFlow, from a mobile dev’s perspective - 19 2015 TF open sourced 2016 TF mobile 2017 TF Lite developer preview 2018 ML Kit 2019 TF Lite exits dev preview TF Mobile to be deprecated ML Kit improves
20. @margaretmz | #MachineLearning #GDE | #AndroidDev Deploy model to Android with TensorFlow Your options: • With ML Kit ◦ (Out of the box) Base APIs ◦ Custom model • Direct deploy to Android ◦ Custom model 20 Custom Models • Download pre trained models or • Train your own model ◦ data ◦ train ◦ convert ◦ inference Note: you can use AutoML to train but no easy implementation on mobile yet
21. @margaretmz | #MachineLearning #GDE | #AndroidDev ML Kit 21 Brings Google’s machine learning expertise to mobile developers in a powerful and easy-to-use package. Powered by TF Lite, hosted on Firebase For custom models, ML kits offers • Dynamic model downloads • A/B testing (via Firebase remote Configuration) • Model compression & conversion (from TensorFlow to TF Lite) Base APIs: Learn more about ML Kit here Image labelling OCR Face detection Barcode scanning Landmark detection Smart reply (coming soon)
22. ML Process for mobile An overview Datasets Train model Convert to TFLite Deploy for inference 22
23. @margaretmz | #MachineLearning #GDE | #AndroidDev Machine Learning process Process: Data, train, convert model, validate converted model, deploy for inference Training vs inference: Example of image classification (supervised ML, computer vision): • Training - feed in labelled data and find a mathematical model that describes the data. • Inference - use the model to predict the label on a new image 23
24. @margaretmz | #MachineLearning #GDE | #AndroidDev Data • Existing datasets ◦ Part of the deep learning framework: ▪ MNIST, CIFAR10, FASHION_MNIST, IMDB movie reviews etc ◦ Open datasets: ▪ MNIST, MS-COCO, IMAGENet, CelebA etc ◦ Kaggle datasets: https://www.kaggle.com/datasets ◦ Google Dataset search tool: https://toolbox.google.com/datasetsearch ◦ TF 2.0: TFDS • Collect your own data 24
25. @margaretmz | #MachineLearning #GDE | #AndroidDev ML models Your options of getting a model for your mobile app: • Download a pre-trained model (here): Inception-v3, mobilenet etc. • Transfer learning with a pre-trained model ◦ Feature extraction or fine tuning on pre-trained model ◦ TensorFlow hub (https://www.tensorflow.org/hub/) • Train your own model from scratch (example in this talk) 25
26. @margaretmz | #MachineLearning #GDE | #AndroidDev Convert, validate & deploy for inference • Convert the model to tflite format • Validate the converted model before deploy • Deploy for inference 26
27. Train a neural network with tf.Keras 27
28. @margaretmz | #MachineLearning #GDE | #AndroidDev MNIST dataset • 60,000 train set and 10,000 test set • 28x28x1 grayscale images • 10 classes: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 • Popular for computer vision ◦ “hello world” tutorial or ◦ benchmarking ML algorithms 28
29. @margaretmz | #MachineLearning #GDE | #AndroidDev Training the model in Colab Launch sample code on Colab → mnist_tfkeras_to_tflite.ipynb 1. Import data 2. Define a model 3. Train a model 4. Save a Keras model & convert to tflite 29
30. @margaretmz | #MachineLearning #GDE | #AndroidDev A typical CNN model architecture MNIST example: • Convolutional layer (definition) • Pooling layer (definition) • Dense (fully-connected layer) definition 30 input conv pool conv pool conv pool Dense 0 1 2 3 4 5 6 7 8 9
31. @margaretmz | #MachineLearning #GDE | #AndroidDev Inspect the model - in python code In python code, after defining the model architecture, use model.summary() to show the model architecture 31
32. @margaretmz | #MachineLearning #GDE | #AndroidDev Virtualize model Use a visualization tool: • TensorBoard • Netron (https://github.com/lutzroeder/Netron) Drop the .tflite model into Netron and see the model visually 32
33. Convert to tflite model 33
34. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow Lite • Exits developer preview soon • Works with Inception & MobileNet • May not support all operations • Supports ◦ Mobile: Android & IOS ◦ Android Things ◦ Raspberry Pi 34
35. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow Lite Converter Convert from Keras model to a tflite model with the tflite converter There are two options: 1. Command line 2. Python API Note: you can convert from GraphDef or SavedModel as well. Read details on tflite converter on TF documentation here 35
36. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite convert through command line To convert a tf.keras model to a tflite model: $ tflite_convert \ $--output_file=mymodel.tflite \ $ --keras_model_file=mymodel.h5 36
37. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite convert through Python code # Create a converter converter = tf.contrib.lite.TFLiteConverter.from_keras_model_file(keras_model) # Set quantize to true converter.post_training_quantize=True # Convert the model tflite_model = converter.convert() # Create the tflite model file tflite_model_name = "mymodel.tflite" open(tflite_model_name, "wb").write(tflite_model) 37
38. @margaretmz | #MachineLearning #GDE | #AndroidDev Validate the tflite model Protip: validate the converted tflite model in python before putting it in Android - # Load TFLite model and allocate tensors. interpreter = tf.contrib.lite.Interpreter(model_path="converted_model.tflite") interpreter.allocate_tensors() # Get input and output tensors. input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() # Test model on random input data. input_shape = input_details[0]['shape'] input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32) interpreter.set_tensor(input_details[0]['index'], input_data) interpreter.invoke() output_data = interpreter.get_tensor(output_details[0]['index']) print(output_data) 38
39. Run tflite on Android 39
40. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite on Android Android sample code DigitRecognizer, step by step: • Place tf.lite model under assets folder • Update build.gradle • Input an image • Data preprocessing • Classify with the model • Post processing • Display result in UI 40
41. @margaretmz | #MachineLearning #GDE | #AndroidDev Dependencies Update build.gradle to include tensorflow lite android { // Make sure model doesn't get compressed when app is compiled aaptOptions { noCompress "tflite" } } dependencies { …. // Add dependency for TensorFlow Lite compile 'org.tensorflow:tensorflow-lite:[version-number]’ } Place the mnist.tflite model file under /assets folder 41
42. @margaretmz | #MachineLearning #GDE | #AndroidDev Input - image data Input to the classifier is an image, your options: • Draw on canvas from custom View • Get image from Gallery or a 3rd party camera • Live frames from Camera2 API Make sure the image dimensions (shape) matches what your classifier expects • 28x28x1- MNIST or FASHION_MNIST gray scale image • 299x299x3 - Inception V3 • 256x256x3 - MobileNet 42
43. @margaretmz | #MachineLearning #GDE | #AndroidDev Image preprocessing • Convert Bitmap to ByteBuffer • Normalize pixel values to be a certain range • Convert from color to grayscale, if needed 43
44. @margaretmz | #MachineLearning #GDE | #AndroidDev Run inference Load the model file located under the assets folder Use the TensorFlow Lite interpreter to run inference on the input image 44
45. @margaretmz | #MachineLearning #GDE | #AndroidDev Post processing The output is an array of probabilities, each correspond to a category Find the category with the highest probability and output result to UI 45
46. @margaretmz | #MachineLearning #GDE | #AndroidDev Summary • Training with tf.Keras is easy • Model conversion to TFLite became easy • Android implementation is still challenging & error-prone: (Hopefully this gets improved in the future!) ◦ Validate tflite model before deploy to Android ◦ Image pre-processing ◦ Input tensor shape? ◦ Color or grayscale? ◦ Post processing 46
47. @margaretmz | #MachineLearning #GDE | #AndroidDev More examples - TFLite demo app Check out Demo app in TensorFlow repo Clone tensorflow project from github git clone https://www.github.com/tensorflow/tensorflow Then open the tflite Android demo from Android Studio /tensorflow/contrib/lite/java/demo Note: TensorFlow Lite moved out contrib as of 10/31/2018 47
48. @margaretmz | #MachineLearning #GDE | #AndroidDev More examples - Inference with GPU TensorFlow Lite Now Faster with Mobile GPUs (Developer Preview) 48
49. @margaretmz | #MachineLearning #GDE | #AndroidDev Thank you! 49 Follow me on Twitter, Medium or GitHub to learn more about Deep learning, TensorFlow and Android @margaretmz @margaretmz margaretmz