tf.Keras to TensorFlow Lite for Android

At DevFest Vancouver 2018, I gave a talk on the end to end process of how to train a model with TensorFlow high level API tf.Keras, obtain the TensorFlow Lite model and deploy it to an Android app.

Transcript

1. @margaretmz tf.Keras to for Android DevFest Vancouver Margaret Maynard-Reid, 10/21/2018
2. @margaretmz | #MachineLearning #GDE | #AndroidDev About me code | speak | write | organize | teach 2
3. @margaretmz | #MachineLearning #GDE | #AndroidDev Topics • Intro ◦ AI, ML, deep learning and computer vision ◦ Learning resources of TensorFlow & Keras • Overview of TensorFlow to Android ◦ Your options for getting a model • Train a neural network with tf.Keras ◦ Deep learning tools ◦ Visualize neural networks • Convert to TensorFlow Lite model ◦ Convert from Keras model to tflite ◦ Inspect & test the tflite model • Run tflite on Android 3
4. Introduction 4
5. @margaretmz | #MachineLearning #GDE | #AndroidDev AI vs. ML vs. Deep Learning Artificial Intelligence Machine Learning Deep Learning: 5 5 - Computer Vision - NLP ….
6. @margaretmz | #MachineLearning #GDE | #AndroidDev Examples of deep learning NSynth - make music with deep learning 6 Computer Vision • Image classification - is this a cat? • Object detection - self driving car • Generating new images Generating new images using generative adversarial networks (GANs) Is this a cat?
7. @margaretmz | #MachineLearning #GDE | #AndroidDev Android vs. Deep Learning 7 1. Language 2. IDE 3. Android SDK
8. @margaretmz | #MachineLearning #GDE | #AndroidDev Android vs. Deep Learning • Deep learning Frameworks: ◦ TensorFlow (>100k stars on Github) ← most popular! ◦ PyTorch ◦ Caffe (1 & 2) ◦ Theano… • Languages: R, Python, Swift, Javascript, Java wrapper • IDE?? • Type of neural networks: ◦ CNN (Convolutional Neural Networks) ◦ RNN (Recurrent Neural Networks) ◦ GAN (Generative Adversarial Networks) ◦ Reinforcement learning 8
9. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow model buillding APIs TensorFlow is a deep learning framework for both research & production Write TensorFlow code in C++, Python, Java, R, Go, SWIFT, JavaScript Deploy to CPU, GPU, TPU, Mobile, Android Things, Raspberry Pi tf.* tf.layers tf.keras Custom Estimator Premade Estimator ← Low level ← mid level ← high level ← model in a box ← distributed execution, tf serving 9
10. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow learning resources Tensorflow.org Deep learning with Python by Francois Chollet TensorFlow on Youtube TensorFlow on Twitter #AskTensorFlow #TensorFlowMeets Collection of interactive ML examples (blogpost | website) TensorFlow Dev Summit Blog.tensorflow.org 10
11. @margaretmz | #MachineLearning #GDE | #AndroidDev Keras vs tf.keras Keras remains an independent open-source project, with these as backend • TensorFlow • Theano • CNTK Tf.keras - part of the TensorFlow core APIs Import tensorflow as tf tf.keras... 11
12. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow on Android - 2015 TensorFlow open sourced - 2016 TensorFlow Mobile - 2017 TensorFlow Lite developer preview - 2018 TensorFlow Lite exits developer preview - 2019 TensorFlow Mobile to be deprecated 12
13. @margaretmz | #MachineLearning #GDE | #AndroidDev Choose a model Here are your options of getting a model, from easy to difficult: • Get a ready-made model through ML Kit • Download a pre-trained model (here): Inception-v3, mobilenet etc. • Use a premade estimator • Use transfer learning with a pre-trained model ◦ Feature extraction or fine tuning on pre-trained model ◦ TensorFlow hub (https://www.tensorflow.org/hub/) • Train your own model from scratch 13
14. @margaretmz | #MachineLearning #GDE | #AndroidDev ML Kit 14 Brings Google’s machine learning expertise to mobile developers in a powerful and easy-to-use package. Powered by TF Lite, with the help of Firebase, ML kits offers: • Dynamic model downloads • A/B testing (via Firebase remote Configuration) • Model compression & conversion (from TensorFlow to TF Lite) Learn more about ML Kit here Image labelling OCR Face detection Barcode scanning Landmark detection Smart reply (coming soon)
15. Train a model from scratch 15
16. @margaretmz | #MachineLearning #GDE | #AndroidDev Tf.keras to tflite to Android • Find a data set • Setup and installations • Write tf.keras code • Convert Keras model to tflite • Run tflite on Android 16
17. @margaretmz | #MachineLearning #GDE | #AndroidDev Finding a dataset Transfer learning or training from scratch, you will need a dataset. • Part of the deep learning framework such as Keras: ◦ MNIST, CIFAR10, FASHION_MNIST, IMDB movie reviews etc • Open datasets: ◦ MNIST, MS-COCO, IMAGENet, CelebA etc • Kaggle datasets • Google Dataset search tool: https://toolbox.google.com/datasetsearch 17
18. @margaretmz | #MachineLearning #GDE | #AndroidDev Tools for deep learning Common tools for deep learning: • Anaconda • Jupyter Notebook • Google Colab • PyCharm 18
19. @margaretmz | #MachineLearning #GDE | #AndroidDev Anaconda, TensorFlow & Keras Why use a virtual environment? Ease of upgrade/downgrade of tensorflow • Download anaconda here • Create a new virtual environment $ conda create -n [my-env-name] • Activate the virtual environment you created $ conda activate [my-env-name] • Install TensorFlow which also contains tf.keras $ pip install --upgrade tensorflow • Launch Jupyter Notebook server $ jupyter notebook My blog post Anaconda, Jupyter Notebook, TensorFlow, Keras 19
20. @margaretmz | #MachineLearning #GDE | #AndroidDev Jupyter Notebook • Interactive programming in the web browser • Great for visualization • Great for collaboration • Popular tool for studying machine learning / deep learning 20
21. @margaretmz | #MachineLearning #GDE | #AndroidDev Google Colab 21 colab.research.google.com/ • Jupyter Notebook running on Google’s VM in the cloud • Free GPU and TPU! • TensorFlow is already installed • Save and share from your Drive • Save directly to GitHub Check out my blogpost on Colab, and TensorFlow team’s blog on Colab
22. @margaretmz | #MachineLearning #GDE | #AndroidDev PyCharm • Made by JetBrains • Similar UI as Android Studio • Run Python scripts in PyCharm ◦ Enable eager execution mode ◦ Set breakpoints ◦ Debug! 22
23. Train a neural network with tf.Keras 23
24. @margaretmz | #MachineLearning #GDE | #AndroidDev Example with Fashion-MNIST Let’s use an example of Fashion-MNIST • Direct drop-in replacement for MNIST • 60,000 train set and 10,000 test set • 28x28x1 grayscale images • 10 classes: T-shirt/top, 1 Trouser , 2 Pullover etc... • Popular for computer vision ◦ “hello world” tutorial or ◦ benchmarking ML algorithms 24
25. @margaretmz | #MachineLearning #GDE | #AndroidDev Training the model in Colab Launch code on Colab → here 1. Import data 2. Define a model 3. Train a model 4. Use the model to predict Note: • we need to first “import tensorflow as tf”, then use tf.keras • not “keras import keras” 25
26. @margaretmz | #MachineLearning #GDE | #AndroidDev A typical CNN model architecture A typical CNN network architecture 26 input conv pool conv pool conv pool Dense
27. Convert to tflite model 27
28. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow Lite • Still in developer preview • Works with Inception & MobileNet • May not support all operations • The future of TensorFlow for ◦ Mobile: Android & IOS ◦ Android Things ◦ Raspberry Pi 28
29. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow Lite Converter Convert from Keras model to a tflite model with the tflite converter There are two options 1. Command line 2. Python API Read the documentation on tflite converter here. 29
30. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite convert through command line To convert a tf.keras model to a tflite model: $ tflite_convert \ $--output_file=mymodel.tflite \ $ --keras_model_file=mymodel.h5 30
31. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite convert through Python code # Create a converter converter = tf.contrib.lite.TocoConverter.from_keras_model_file(model_file) # Set quantize to true converter.post_training_quantize=True # Convert the model tflite_model = converter.convert() # Create the tflite model file tflite_model_name = "mymodel.tflite" open(tflite_model_name, "wb").write(tflite_model) 31
32. @margaretmz | #MachineLearning #GDE | #AndroidDev Inspect the model • Use model.summary() shows the model architecture • Use a visualization tool: ◦ TensorBoard ◦ Netron (https://github.com/lutzroeder/Netron) 32
33. @margaretmz | #MachineLearning #GDE | #AndroidDev Test the tflite model Protip: testing the tflite model in python before putting it in Android - # Load TFLite model and allocate tensors. interpreter = tf.contrib.lite.Interpreter(model_path="converted_model.tflite") interpreter.allocate_tensors() # Get input and output tensors. input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() # Test model on random input data. input_shape = input_details[0]['shape'] input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32) interpreter.set_tensor(input_details[0]['index'], input_data) interpreter.invoke() output_data = interpreter.get_tensor(output_details[0]['index']) print(output_data) 33
34. Run tflite on Android 34
35. @margaretmz | #MachineLearning #GDE | #AndroidDev Android tflite demo app Clone tensorflow project from github git clone https://www.github.com/tensorflow/tensorflow Then open the tflite Android demo from Android Studio /tensorflow/contrib/lite/java/demo Note: TensorFlow Lite moved out contrib as of 10/31/2018 35
36. @margaretmz | #MachineLearning #GDE | #AndroidDev The Android app In build.gradle to add dependency of the TensoFlow lite wrapper compile ‘org.tensorflow:tensorflow-lite:+’ Place the .tflite model file under /assets folder Update the Classifier Java class 36
37. @margaretmz | #MachineLearning #GDE | #AndroidDev Watchouts • Image pre-processing • Input tensor shape • Color or grayscale? • Test tflite model before putting it in Android 37
38. @margaretmz | #MachineLearning #GDE | #AndroidDev Thank you! 38 Follow me on Twitter, Medium or GitHub to learn more about Deep learning, TensorFlow and Android @margaretmz @margaretmz margaretmz