Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Machine Learning for Mobile with TensorFlow

Machine Learning for Mobile with TensorFlow

I gave a talk for GDG Boulder:
https://www.meetup.com/Google-Developer-Group-Boulder/events/257013855/

This talk shares how to make intelligent apps with TensorFlow. I gave an overview of the various options from using ready made APIs to training your own custom models. I walked through the end to end process of how to train a model with TensorFlow high level API tf.Keras, convert to a TensorFlow Lite model and deploy it to an Android app.

Margaret Maynard-Reid

January 29, 2019
Tweet

More Decks by Margaret Maynard-Reid

Other Decks in Technology

Transcript

  1. @margaretmz | #MachineLearning #GDE | #AndroidDev Slides Slides for this

    talk is on speakerdeck bit.ly/dl-for-android-with-tf-devfestsv19 Click on download PDF to access the links in downloaded file 2
  2. @margaretmz | #MachineLearning #GDE | #AndroidDev Topics • Intro ◦

    TensorFlow, tf.Keras, tools • ML on Android, your options ◦ Hosted with ML Kit ◦ Direct deploy to Android app • ML process for mobile: data -> train -> convert -> deploy for inference • End to end tf.Keras to TF Lite to Android ◦ Train a model from scratch with tf.Keras ◦ Convert Keras model to tflite ◦ Inspect & test the tflite model ◦ Run tflite on Android 3
  3. @margaretmz | #MachineLearning #GDE | #AndroidDev AI vs. ML vs.

    Deep Learning Artificial Intelligence Machine Learning Deep Learning: 5 5 - Computer Vision - NLP ….
  4. @margaretmz | #MachineLearning #GDE | #AndroidDev Some examples of deep

    learning NSynth - make music with deep learning 6 Computer Vision • Image classification - is this a cat? • Object detection - self driving car • Generating new images Generating new images using generative adversarial networks (GANs) Is this a cat?
  5. @margaretmz | #MachineLearning #GDE | #AndroidDev Get started with Deep

    Learning • Deep learning Frameworks: ◦ TensorFlow (>100k stars on Github) ← most popular! ◦ PyTorch ◦ Caffe (1 & 2) ◦ Theano… • Languages: Python, Swift, Javascript etc. • IDE - Colab or PyCharm (for Python code) • Type of neural networks: ◦ CNN (Convolutional Neural Networks) ◦ RNN (Recurrent Neural Networks) ◦ GAN (Generative Adversarial Networks) ◦ ... 7
  6. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow model building APIs

    TensorFlow is a deep learning framework for both research & production Write TensorFlow code in C++, Python, Java, R, Go, SWIFT, JavaScript Deploy to CPU, GPU, TPU, Mobile, Android Things, Raspberry Pi tf.* tf.layers tf.keras Custom Estimator Premade Estimator ← Low level ← Mid level (moving to tf.keras in TF 2.0) ← High level ← Model in a box ← Distributed execution, tf serving 8
  7. @margaretmz | #MachineLearning #GDE | #AndroidDev tf.Keras vs Keras tf.keras

    - part of the TensorFlow core APIs import tensorflow as tf keras = tf.keras... Keras remains an independent open-source project, with backend: • TensorFlow (Note: you should just use tf.keras, instead of Keras + TF as backend) • Theano • CNTK... 9
  8. @margaretmz | #MachineLearning #GDE | #AndroidDev tf.Keras model building APIs

    • Sequential - the easiest way • Functional - more flexibility • Model subclassing - extend a Model class Learn more in Josh Gordon’s blog: What are Symbolic and Imperative APIs in TensorFlow 2.0? 10
  9. @margaretmz | #MachineLearning #GDE | #AndroidDev Blog.tensorflow.org TensorFlow learning resources

    Tensorflow.org Deep learning with Python by Francois Chollet TensorFlow on Youtube TensorFlow on Twitter #AskTensorFlow #TensorFlowMeets Collection of interactive ML examples (blogpost | website) 11 Interested in learning about TensorFlow 2.0 preview and try it out? Read My Notes on TensorFlow 2.0 TensorFlow Dev Summit
  10. @margaretmz | #MachineLearning #GDE | #AndroidDev Tools for deep learning

    • Anaconda or Virtualenv - for creating virtual environments • Jupyter Notebook • Google Colab • PyCharm 13
  11. @margaretmz | #MachineLearning #GDE | #AndroidDev Anaconda, TensorFlow & Keras

    Why use a virtual environment? Ease of upgrade/downgrade of tensorflow • Download anaconda here • Create a new virtual environment $ conda create -n [my-env-name] • Activate the virtual environment you created $ conda activate [my-env-name] • Install TensorFlow which also contains tf.keras $ pip install --upgrade tensorflow • Launch Jupyter Notebook server $ jupyter notebook My blog post Anaconda, Jupyter Notebook, TensorFlow, Keras 14
  12. @margaretmz | #MachineLearning #GDE | #AndroidDev Jupyter Notebook • Interactive

    programming in the web browser • Great for visualization • Great for collaboration • Popular tool for studying machine learning / deep learning 15
  13. @margaretmz | #MachineLearning #GDE | #AndroidDev Google Colab What is

    Google Colab? • Jupyter Notebook ◦ stored on Google Drive ◦ running on Google’s VM in the cloud • Free GPU and TPU! • TensorFlow is already installed • Save and share from your Drive • Save directly to GitHub 16 Check out these learning resources • My blog on Colab • TF team’s blog on Colab • Laurence’ Video Build a deep neural network in 4 mins with TensorFlow in Colab • Paige’s video How to take advantage of GPUs & TPUs for your ML project • Sam’s blog Keras on TPUs in Colab Launch Colab from colab.research.google.com/
  14. @margaretmz | #MachineLearning #GDE | #AndroidDev PyCharm • Made by

    JetBrains • Similar UI as Android Studio • Run Python scripts in PyCharm ◦ Enable eager execution mode (default in TensorFlow 2.0) ◦ Set breakpoints ◦ Debug! 17
  15. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow for mobile apps

    History of TensorFlow, from a mobile dev’s perspective - 19 2015 TF open sourced 2016 TF mobile 2017 TF Lite developer preview 2018 ML Kit 2019 TF Lite exits dev preview TF Mobile to be deprecated ML Kit improves
  16. @margaretmz | #MachineLearning #GDE | #AndroidDev Deploy model to Android

    with TensorFlow Your options: • With ML Kit ◦ (Out of the box) Base APIs ◦ Custom model • Direct deploy to Android ◦ Custom model 20 Custom Models • Download pre trained models or • Train your own model ◦ data ◦ train ◦ convert ◦ inference Note: you can use AutoML to train but no easy implementation on mobile yet
  17. @margaretmz | #MachineLearning #GDE | #AndroidDev ML Kit 21 Brings

    Google’s machine learning expertise to mobile developers in a powerful and easy-to-use package. Powered by TF Lite, hosted on Firebase For custom models, ML kits offers • Dynamic model downloads • A/B testing (via Firebase remote Configuration) • Model compression & conversion (from TensorFlow to TF Lite) Base APIs: Learn more about ML Kit here Image labelling OCR Face detection Barcode scanning Landmark detection Smart reply (coming soon)
  18. @margaretmz | #MachineLearning #GDE | #AndroidDev Machine Learning process Process:

    Data, train, convert model, validate converted model, deploy for inference Training vs inference: Example of image classification (supervised ML, computer vision): • Training - feed in labelled data and find a mathematical model that describes the data. • Inference - use the model to predict the label on a new image 23
  19. @margaretmz | #MachineLearning #GDE | #AndroidDev Data • Existing datasets

    ◦ Part of the deep learning framework: ▪ MNIST, CIFAR10, FASHION_MNIST, IMDB movie reviews etc ◦ Open datasets: ▪ MNIST, MS-COCO, IMAGENet, CelebA etc ◦ Kaggle datasets: https://www.kaggle.com/datasets ◦ Google Dataset search tool: https://toolbox.google.com/datasetsearch ◦ TF 2.0: TFDS • Collect your own data 24
  20. @margaretmz | #MachineLearning #GDE | #AndroidDev ML models Your options

    of getting a model for your mobile app: • Download a pre-trained model (here): Inception-v3, mobilenet etc. • Transfer learning with a pre-trained model ◦ Feature extraction or fine tuning on pre-trained model ◦ TensorFlow hub (https://www.tensorflow.org/hub/) • Train your own model from scratch (example in this talk) 25
  21. @margaretmz | #MachineLearning #GDE | #AndroidDev Convert, validate & deploy

    for inference • Convert the model to tflite format • Validate the converted model before deploy • Deploy for inference 26
  22. @margaretmz | #MachineLearning #GDE | #AndroidDev MNIST dataset • 60,000

    train set and 10,000 test set • 28x28x1 grayscale images • 10 classes: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 • Popular for computer vision ◦ “hello world” tutorial or ◦ benchmarking ML algorithms 28
  23. @margaretmz | #MachineLearning #GDE | #AndroidDev Training the model in

    Colab Launch sample code on Colab → mnist_tfkeras_to_tflite.ipynb 1. Import data 2. Define a model 3. Train a model 4. Save a Keras model & convert to tflite 29
  24. @margaretmz | #MachineLearning #GDE | #AndroidDev A typical CNN model

    architecture MNIST example: • Convolutional layer (definition) • Pooling layer (definition) • Dense (fully-connected layer) definition 30 input conv pool conv pool conv pool Dense 0 1 2 3 4 5 6 7 8 9
  25. @margaretmz | #MachineLearning #GDE | #AndroidDev Inspect the model -

    in python code In python code, after defining the model architecture, use model.summary() to show the model architecture 31
  26. @margaretmz | #MachineLearning #GDE | #AndroidDev Virtualize model Use a

    visualization tool: • TensorBoard • Netron (https://github.com/lutzroeder/Netron) Drop the .tflite model into Netron and see the model visually 32
  27. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow Lite • Exits

    developer preview soon • Works with Inception & MobileNet • May not support all operations • Supports ◦ Mobile: Android & IOS ◦ Android Things ◦ Raspberry Pi 34
  28. @margaretmz | #MachineLearning #GDE | #AndroidDev TensorFlow Lite Converter Convert

    from Keras model to a tflite model with the tflite converter There are two options: 1. Command line 2. Python API Note: you can convert from GraphDef or SavedModel as well. Read details on tflite converter on TF documentation here 35
  29. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite convert through command

    line To convert a tf.keras model to a tflite model: $ tflite_convert \ $--output_file=mymodel.tflite \ $ --keras_model_file=mymodel.h5 36
  30. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite convert through Python

    code # Create a converter converter = tf.contrib.lite.TFLiteConverter.from_keras_model_file(keras_model) # Set quantize to true converter.post_training_quantize=True # Convert the model tflite_model = converter.convert() # Create the tflite model file tflite_model_name = "mymodel.tflite" open(tflite_model_name, "wb").write(tflite_model) 37
  31. @margaretmz | #MachineLearning #GDE | #AndroidDev Validate the tflite model

    Protip: validate the converted tflite model in python before putting it in Android - # Load TFLite model and allocate tensors. interpreter = tf.contrib.lite.Interpreter(model_path="converted_model.tflite") interpreter.allocate_tensors() # Get input and output tensors. input_details = interpreter.get_input_details() output_details = interpreter.get_output_details() # Test model on random input data. input_shape = input_details[0]['shape'] input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32) interpreter.set_tensor(input_details[0]['index'], input_data) interpreter.invoke() output_data = interpreter.get_tensor(output_details[0]['index']) print(output_data) 38
  32. @margaretmz | #MachineLearning #GDE | #AndroidDev Tflite on Android Android

    sample code DigitRecognizer, step by step: • Place tf.lite model under assets folder • Update build.gradle • Input an image • Data preprocessing • Classify with the model • Post processing • Display result in UI 40
  33. @margaretmz | #MachineLearning #GDE | #AndroidDev Dependencies Update build.gradle to

    include tensorflow lite android { // Make sure model doesn't get compressed when app is compiled aaptOptions { noCompress "tflite" } } dependencies { …. // Add dependency for TensorFlow Lite compile 'org.tensorflow:tensorflow-lite:[version-number]’ } Place the mnist.tflite model file under /assets folder 41
  34. @margaretmz | #MachineLearning #GDE | #AndroidDev Input - image data

    Input to the classifier is an image, your options: • Draw on canvas from custom View • Get image from Gallery or a 3rd party camera • Live frames from Camera2 API Make sure the image dimensions (shape) matches what your classifier expects • 28x28x1- MNIST or FASHION_MNIST gray scale image • 299x299x3 - Inception V3 • 256x256x3 - MobileNet 42
  35. @margaretmz | #MachineLearning #GDE | #AndroidDev Image preprocessing • Convert

    Bitmap to ByteBuffer • Normalize pixel values to be a certain range • Convert from color to grayscale, if needed 43
  36. @margaretmz | #MachineLearning #GDE | #AndroidDev Run inference Load the

    model file located under the assets folder Use the TensorFlow Lite interpreter to run inference on the input image 44
  37. @margaretmz | #MachineLearning #GDE | #AndroidDev Post processing The output

    is an array of probabilities, each correspond to a category Find the category with the highest probability and output result to UI 45
  38. @margaretmz | #MachineLearning #GDE | #AndroidDev Summary • Training with

    tf.Keras is easy • Model conversion to TFLite became easy • Android implementation is still challenging & error-prone: (Hopefully this gets improved in the future!) ◦ Validate tflite model before deploy to Android ◦ Image pre-processing ◦ Input tensor shape? ◦ Color or grayscale? ◦ Post processing 46
  39. @margaretmz | #MachineLearning #GDE | #AndroidDev More examples - TFLite

    demo app Check out Demo app in TensorFlow repo Clone tensorflow project from github git clone https://www.github.com/tensorflow/tensorflow Then open the tflite Android demo from Android Studio /tensorflow/contrib/lite/java/demo Note: TensorFlow Lite moved out contrib as of 10/31/2018 47
  40. @margaretmz | #MachineLearning #GDE | #AndroidDev More examples - Inference

    with GPU TensorFlow Lite Now Faster with Mobile GPUs (Developer Preview) 48
  41. @margaretmz | #MachineLearning #GDE | #AndroidDev Thank you! 49 Follow

    me on Twitter, Medium or GitHub to learn more about Deep learning, TensorFlow and Android @margaretmz @margaretmz margaretmz