From TensorFlow to ML Kit: Power your Android application with machine learning

7b5a07956eb0b62be7214d043821a987?s=47 jinqian
February 08, 2019

From TensorFlow to ML Kit: Power your Android application with machine learning

Presented at DroidKaigi 2019

Video: https://www.youtube.com/watch?v=EZgbD_9wdiY

7b5a07956eb0b62be7214d043821a987?s=128

jinqian

February 08, 2019
Tweet

Transcript

  1. From TensorFlow to ML Kit: Power your Android application with

    machine learning DroidKaigi 2019 | Qian Jin ⾦金金芊 | @bonbonking Image Credit: https://unsplash.com/photos/pP7EgaYDRKg
  2. ⾦金金芊, Androidデベロッパーです, GDE IoT 杭州からきました! パリ在住 " @bonbonking

  3. Cloud Intelligence Image Credit: Berndnaut Smilde

  4. On-device Intelligence Image Credit: https://unsplash.com/photos/93n4PZzzlNk

  5. !5

  6. Source !6 Source: https://9to5google.com/2017/01/10/qualcomm-snapdragon-835-machine-learning-tensorflow/

  7. AI at the Edge Source: https://cloud.google.com/iot-edge/, https://cloud.google.com/edge-tpu/, https://en.wikipedia.org/wiki/Tensor_processing_unit !7 TPU:

    Tensor processing unit
  8. Machine Learning — Mobile Development !8 Nov 2015 TensorFlow initial

    release Sep 2016 TensorFlow Mobile (Android & iOS demo) May 2017 Annonce of TF Lite at Google I/O June 2017 MobileNet v1 Apr 2018 MobileNet v2 May 2018 Annonce of ML Kit at Google I/O Nov 2017 TF Lite Developer Preview June 2018 WWDC Core ML 2 Create ML June 2017 WWDC Core ML Vision API NLP June 2016 WWDC Speech API May 2016 Google I/O Mobile Vision API
  9. !9 Sep 2016 TensorFlow Mobile (Android & iOS demo)

  10. TensorFlow <3 Android Image Credit: https://unsplash.com/photos/-9INjxHfZak

  11. !11

  12. MACHINE LEARNING ALL THE THINGS!

  13. Magritte Ceci n’est pas une pomme.

  14. !14

  15. I THOUGHT THERE WERE MODELS FOR EVERYTHING...

  16. Neural Network in a Nutshell Credit: Image Credit: https://unsplash.com/photos/BTgABQwq7HI

  17. Here’s a Neuron Reference: https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html#weights !17 :)

  18. With Synapses Reference: https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html#weights !18 :)

  19. Here are 3 layers of neurons Reference: https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html#weights !19 w1

    w2 w3 H1 I1 I2 I3 O1 O2 Input Layer Hidden Layer Output Layer
  20. Here’s a Neural Network !20

  21. Inference: Prediction on an image !21

  22. Inference: Prediction on an image !22

  23. Inference: Prediction on an image !23 Apple: 0.98 Banana: 0.02

  24. !24

  25. Back Propagation !25

  26. Back Propagation !26 Apple: 0.34 Banana: 0.66

  27. Apple: 0.34 Banana: 0.66 Back Propagation !27 Prediction Error

  28. Apple: 0.34 Banana: 0.66 Back Propagation !28 Prediction Error

  29. Apple: 0.34 Banana: 0.66 Back Propagation !29 Prediction Error

  30. Back Propagation !30 Apple: 0.87 Banana: 0.13

  31. Back Propagation !31 Banana: 0.93 Apple: 0.07

  32. Deep Convolutional Neural Network !32 Image Credit: https://github.com/tensorflow/models/tree/master/research/inception Visualisation of

    Inception v3 Model Architecture Edges Shapes High Level Features Classifiers
  33. Source: CS231n Convolutional Neural Networks for Visual Recognition http://cs231n.stanford.edu/ !33

  34. Source: https://code.facebook.com/posts/1687861518126048/facebook-to-open-source-ai-hardware-design/ !34

  35. Transfer Learning !35 Keep all weights identical except these ones

  36. Build Magritte prototype Credit: https://unsplash.com/photos/loAgTdeDcIU

  37. Image Credit: https://xkcd.com/1987/ !37

  38. Image Credit: https://xkcd.com/1838/ !38

  39. Gather Training Data !39

  40. !40 Retrain a Model python -m tensorflow/examples/image_retraining/retrain.py \ --how_many_training_steps=500 \

    --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/ \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --image_dir=tf_files/fruit_photos
  41. Overfitting

  42. None
  43. Obtain the Retrained Model !43 model.pb label.txt apple banana grape

    kiwi orange pineapple strawberry watermelon lemon
  44. App size ~80MB

  45. Why is model size this heavy? All weights are stored

    as they are (32-bit floats) !45
  46. Post-training Quantization Source: https://www.tensorflow.org/performance/post_training_quantization https://cloud.google.com/blog/products/gcp/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu 32 bit float => 8

    bit integer ~80MB => ~20MB !46
  47. Version: April 2017 Device: Nexus 5X Average inference time: 3-4s

  48. None
  49. !49 June 2017 MobileNet v1

  50. MobileNetV1 Mobile-first computer vision models for TensorFlow !50 Image credit

    : https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md
  51. Model size is reduced from ~20Mb to ~5Mb Source: https://research.googleblog.com/2017/06/mobilenets-open-source-models-for.html

  52. !52 Optimize for Mobile > IMAGE_SIZE=224 > ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}"

  53. !53 Optimize for Mobile python -m tensorflow/examples/image_retraining/retrain.py \ --bottleneck_dir=tf_files/bottlenecks \

    --how_many_training_steps=500 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --architecture="${ARCHITECTURE}" \ —image_dir=tf_files/fruit_photos
  54. Version: November 2017 Device: Nexus 5X Average inference time: 1s

  55. None
  56. None
  57. Model Inception V3 Quantized

  58. Model MobileNets_1.0_224

  59. Model MobileNets_0.5_224

  60. Model MobileNets_0.25_224

  61. Underneath the hood

  62. Image Sampling Get Image from Camera Preview Crop the center

    square Resize Sample Image !62
  63. Reference: https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/ !63 Android SDK (Java) Android NDK (C++) Classifier

    Implementation TensorFlow JNI wrapper Image (Bitmap) Trained Model top_results Classifications + Confidence input_tensor 1 2 3 4 Camera Preview Overlay Display
  64. Android FilesDir model.pb Labels model.pb label.txt

  65. !65 May 2018 Annonce of ML Kit at Google I/O

  66. ML Kit is in da house

  67. TensorFlow Cloud Machine Learning Engine Ready to use Machine Learning

    API Use your own data to train models Cloud Vision API Cloud Speech API Cloud Translation API Cloud Natural Language API Cloud Video Intelligence API
  68. What’s ML Kit for Firebase? !68 ML Kit Vision APIs

    ML Kit Custom Models
  69. What’s ML Kit for Firebase? !69 Mobile Vision API TensorFlow

    Lite Android Neural Network API Google Cloud Vision API + + ML Kit Vision APIs ML Kit Custom Models
  70. Natural Language Identify the language of text with ML Kit

  71. Vision APIs

  72. None
  73. None
  74. None
  75. Source: https://firebase.google.com/docs/ml-kit/detect-faces !75 Face Contour Detection

  76. None
  77. None
  78. None
  79. Custom Model

  80. TensorFlow lite model hosting On-device ML inference Automatic model fallback

    Over-the-air model updates !80
  81. Reference: https://www.tensorflow.org/lite/convert/cmdline_examples !81 Convert model to TF Lite (model.tflite) Host

    your TF Lite model on Firebase Use the TF Lite model for inference Train your TF model (model.pb) GraphDef SavedModel tf.Keras model
  82. !82 tflite_convert (post TF 1.9 float type) tflite_convert \ --graph_def_file=/tmp/magritte_retrained_graph.pb

    \ --output_file=/tmp/magritte_graph.tflite \ --inference_type=FLOAT \ --input_shape=1,224,224,3 \ --input_array=input \ --output_array=final_result \ --default_ranges_min=0 \ --default_ranges_max=6 *2 other input types: --saved_model_dir & --keras_model_file
  83. !83 tflite_convert (post TF 1.9 float type) tflite_convert \ --graph_def_file=/tmp/magritte_retrained_graph.pb

    \ --output_file=/tmp/magritte_graph.tflite \ --inference_type=FLOAT \ --input_shape=1,224,224,3 \ --input_array=input \ --output_array=final_result \ --default_ranges_min=0 \ --default_ranges_max=6 *2 other input types: --saved_model_dir & --keras_model_file Reference: https://www.tensorflow.org/lite/convert/cmdline_reference
  84. !84 tflite_convert (post TF 1.9 quantized type) tflite_convert \ --graph_def_file=/tmp/magritte_quantized_graph.pb

    \ --output_file=/tmp/magritte_graph.tflite \ --inference_type=QUANTIZED_UINT8 \ --input_shape=1,224,224,3 \ --input_array=input \ --output_array=final_result_fruits \ --default_ranges_min=0 \ --default_ranges_max=6 \ --mean_value=128 \ --std_value=128 \ * mean_vale & std_value are only needed if inference_input_type is QUANTIZED_UINT8. Reference: https://www.tensorflow.org/lite/convert/cmdline_reference
  85. Obtain the TF Lite model !85 model.tflite label.txt apple banana

    grape kiwi orange pineapple strawberry watermelon lemon
  86. None
  87. FirebaseModelInterpreter FirebaseModelInputs FirebaseModelOutputs INPUT OUTPUT

  88. !88 Input Dimensions & Constants companion object { private const

    val HOSTED_MODEL_NAME = "magritte" private const val LOCAL_MODEL_NAME = "magritte" private const val LOCAL_MODEL_PATH = "magritte.tflite" private const val LABEL_PATH = "magritte_labels.txt" const val DIM_BATCH_SIZE = 1 const val DIM_PIXEL_SIZE = 3 const val DIM_IMG_SIZE_X = 224 const val DIM_IMG_SIZE_Y = 224 private const val MEAN = 128 private const val STD = 128.0f }
  89. !89 Set up custom model classifier val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME)

    .setAssetFilePath(LOCAL_MODEL_ASSET).build() val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
  90. !90 Create local model source val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build()

    val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
  91. !91 Create cloud model source val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build()

    val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
  92. !92 Register model sources val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build() val

    cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
  93. !93 Set up FirebaseModelInterpreter val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build() val

    cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
  94. !94 Input Dimensions // input & output options for non-quantized

    model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
  95. !95 Output Dimensions // input & output options for non-quantized

    model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
  96. !96 Input/Output Options // input & output options for non-quantized

    model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
  97. !97 Input/Output Options // input & output options for non-quantized

    model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
  98. !98 Convert bitmap to ByteBuffer private val intValues = IntArray(DIM_IMG_SIZE_X

    * DIM_IMG_SIZE_Y) private val imgData = ByteBuffer.allocateDirect( 4 * DIM_BATCH_SIZE * DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE) @Synchronized private fun convertBitmapToByteBuffer(bitmap: Bitmap): ByteBuffer { imgData.apply { order(ByteOrder.nativeOrder()) rewind() } bitmap.getPixels(intValues, 0, bitmap.width, 0, 0, bitmap.width, bitmap.height) // Preprocess the image data to normalized float intValues.forEach { imgData.putFloat(((it shr 16 and 0xFF) - MEAN) / STD) imgData.putFloat(((it shr 8 and 0xFF) - MEAN) / STD) imgData.putFloat(((it and 0xFF) - MEAN) / STD) } return imgData }
  99. !99 Convert bitmap to ByteBuffer private val intValues = IntArray(DIM_IMG_SIZE_X

    * DIM_IMG_SIZE_Y) private val imgData = ByteBuffer.allocateDirect( 4 * DIM_BATCH_SIZE * DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE) @Synchronized private fun convertBitmapToByteBuffer(bitmap: Bitmap): ByteBuffer { imgData.apply { order(ByteOrder.nativeOrder()) rewind() } bitmap.getPixels(intValues, 0, bitmap.width, 0, 0, bitmap.width, bitmap.height) // Preprocess the image data to normalized float intValues.forEach { imgData.putFloat(((it shr 16 and 0xFF) - MEAN) / STD) imgData.putFloat(((it shr 8 and 0xFF) - MEAN) / STD) imgData.putFloat(((it and 0xFF) - MEAN) / STD) } return imgData }
  100. Normalized ByteBuffer FirebaseModelInputs INPUT

  101. !101 Run the inference override fun process(bitmap: Bitmap) { val

    imageByteBuffer = convertBitmapToByteBuffer(bitmap) val inputs = FirebaseModelInputs.Builder().add(imageByteBuffer).build() interpreter?.run(inputs, inputOutputOptions) ?.addOnSuccessListener { val labelProbArray = it.getOutput<Array<FloatArray>>(0) val results = getTopLabel(labelProbArray) // … } } private fun getTopLabel(labelProbArray: Array<FloatArray>): Pair<String, Float> { return labelList.asSequence().mapIndexed { i, label -> Pair(label, labelProbArray[0][i]) }.sortedBy { it.second }.last() }
  102. FirebaseModelOutputs OUTPUT apple banana grape kiwi orange pineapple strawberry watermelon

    lemon
  103. Model MobileNets_1.0_224

  104. Model MobileNets_1.0_224

  105. Play Rock-Paper-Scissors-Spock-Lizard with your Android Things

  106. None
  107. Applications Node.js + Express Firebase (Hosting/Function/Storage) Android Things (Peripheral I/O

    APIs) Android SDK (TextToSpeech API/Camera API) TensorFlow Lite MobileNetV1 !107
  108. Data Collector App !108

  109. Collected Photos

  110. PWM (Pulse Width Modulation) & Servo Motor !110

  111. 16-Channel PWM Servo Driver !111

  112. Android Things Starter Kit - NXP i.MX7D !112

  113. !113

  114. None
  115. None
  116. Source: !116

  117. !117

  118. Total Time Spent An entire weekend (of burning finger and

    slapped face)! My robot doesn't respect (yet) Asimov's Three Laws of Robotics !118
  119. Take-away Image Credit: https://unsplash.com/photos/Kj2SaNHG-hg

  120. TensorFlow & MobileNet TensorFlow Mobile (deprecated) . TensorFlow Lite /

    • Performance optimized for mobile and embedded devices • Tools support wider range of model formats conversion From MobileNetV1 to MobileNetV2 !120
  121. Firebase ML Kit Still in Beta so API may break

    No callback for model downloading (yet) More features are coming Give feedbacks! !121
  122. Other Trends Model Security How to protect your model? Cloud

    + Edge End-to-end AI infrastructure Machine Learning pipelines (e.g. KubeFlow) !122
  123. Federate Learning Collaborative Machine Learning without Centralized Training Data !123

    Source: https://research.googleblog.com/2017/04/federated-learning-collaborative.html
  124. Google Translate for Sign Language !124

  125. Thank you! ありがとうございました! Magritte: https://github.com/xebia-france/magritte ML Kit in Action: https://github.com/jinqian/MLKit-in-actions

    Hand Game Robot: https://www.hackster.io/bonbonking/ Twitter: @bonbonking
  126. Resources • Artificial neural network: https://en.wikipedia.org/wiki/Artificial_neural_network • Deep Learning: https://en.wikipedia.org/wiki/Deep_learning

    • Convolutional Neural Network: https://en.wikipedia.org/wiki/Convolutional_neural_network • TensorFlow for Poets: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ • TensorFlow for Poets 2: Optimize for Mobile: https://codelabs.developers.google.com/codelabs/tensorflow- for-poets-2/ • TensorFlow Glossary: https://www.tensorflow.org/versions/r0.12/resources/glossary • Talk Magritte for DroidCon London: https://speakerdeck.com/jinqian/droidcon-london-heat-the-neurons- of-your-smartphone-with-deep-learning • Medium article: Android meets Machine Learning https://medium.com/xebia-france/android-meets- machine-learning-part-1-from-tensorflow-mobile-lite-to-ml-kit-4c7e6bc8eee3 !126