From TensorFlow to ML Kit: Power your Android application with machine learning

Presented at DroidKaigi 2019

Video: https://www.youtube.com/watch?v=EZgbD_9wdiY

Transcript

1. From TensorFlow to ML Kit: Power your Android application with machine learning DroidKaigi 2019 | Qian Jin ⾦金金芊 | @bonbonking Image Credit: https://unsplash.com/photos/pP7EgaYDRKg
2. ⾦金金芊, Androidデベロッパーです, GDE IoT 杭州からきました! パリ在住 " @bonbonking
3. Cloud Intelligence Image Credit: Berndnaut Smilde
4. On-device Intelligence Image Credit: https://unsplash.com/photos/93n4PZzzlNk
5. !5
6. Source !6 Source: https://9to5google.com/2017/01/10/qualcomm-snapdragon-835-machine-learning-tensorflow/
7. AI at the Edge Source: https://cloud.google.com/iot-edge/, https://cloud.google.com/edge-tpu/, https://en.wikipedia.org/wiki/Tensor_processing_unit !7 TPU: Tensor processing unit
8. Machine Learning — Mobile Development !8 Nov 2015 TensorFlow initial release Sep 2016 TensorFlow Mobile (Android & iOS demo) May 2017 Annonce of TF Lite at Google I/O June 2017 MobileNet v1 Apr 2018 MobileNet v2 May 2018 Annonce of ML Kit at Google I/O Nov 2017 TF Lite Developer Preview June 2018 WWDC Core ML 2 Create ML June 2017 WWDC Core ML Vision API NLP June 2016 WWDC Speech API May 2016 Google I/O Mobile Vision API
9. !9 Sep 2016 TensorFlow Mobile (Android & iOS demo)
10. TensorFlow <3 Android Image Credit: https://unsplash.com/photos/-9INjxHfZak
11. !11
12. MACHINE LEARNING ALL THE THINGS!
13. Magritte Ceci n’est pas une pomme.
14. !14
15. I THOUGHT THERE WERE MODELS FOR EVERYTHING...
16. Neural Network in a Nutshell Credit: Image Credit: https://unsplash.com/photos/BTgABQwq7HI
17. Here’s a Neuron Reference: https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html#weights !17 :)
18. With Synapses Reference: https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html#weights !18 :)
19. Here are 3 layers of neurons Reference: https://ml-cheatsheet.readthedocs.io/en/latest/nn_concepts.html#weights !19 w1 w2 w3 H1 I1 I2 I3 O1 O2 Input Layer Hidden Layer Output Layer
20. Here’s a Neural Network !20
21. Inference: Prediction on an image !21
22. Inference: Prediction on an image !22
23. Inference: Prediction on an image !23 Apple: 0.98 Banana: 0.02
24. !24
25. Back Propagation !25
26. Back Propagation !26 Apple: 0.34 Banana: 0.66
27. Apple: 0.34 Banana: 0.66 Back Propagation !27 Prediction Error
28. Apple: 0.34 Banana: 0.66 Back Propagation !28 Prediction Error
29. Apple: 0.34 Banana: 0.66 Back Propagation !29 Prediction Error
30. Back Propagation !30 Apple: 0.87 Banana: 0.13
31. Back Propagation !31 Banana: 0.93 Apple: 0.07
32. Deep Convolutional Neural Network !32 Image Credit: https://github.com/tensorflow/models/tree/master/research/inception Visualisation of Inception v3 Model Architecture Edges Shapes High Level Features Classifiers
33. Source: CS231n Convolutional Neural Networks for Visual Recognition http://cs231n.stanford.edu/ !33
34. Source: https://code.facebook.com/posts/1687861518126048/facebook-to-open-source-ai-hardware-design/ !34
35. Transfer Learning !35 Keep all weights identical except these ones
36. Build Magritte prototype Credit: https://unsplash.com/photos/loAgTdeDcIU
37. Image Credit: https://xkcd.com/1987/ !37
38. Image Credit: https://xkcd.com/1838/ !38
39. Gather Training Data !39
40. !40 Retrain a Model python -m tensorflow/examples/image_retraining/retrain.py \ --how_many_training_steps=500 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/ \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --image_dir=tf_files/fruit_photos
41. Overfitting
43. Obtain the Retrained Model !43 model.pb label.txt apple banana grape kiwi orange pineapple strawberry watermelon lemon
44. App size ~80MB
45. Why is model size this heavy? All weights are stored as they are (32-bit floats) !45
46. Post-training Quantization Source: https://www.tensorflow.org/performance/post_training_quantization https://cloud.google.com/blog/products/gcp/an-in-depth-look-at-googles-first-tensor-processing-unit-tpu 32 bit float => 8 bit integer ~80MB => ~20MB !46
47. Version: April 2017 Device: Nexus 5X Average inference time: 3-4s
49. !49 June 2017 MobileNet v1
50. MobileNetV1 Mobile-first computer vision models for TensorFlow !50 Image credit : https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet_v1.md
51. Model size is reduced from ~20Mb to ~5Mb Source: https://research.googleblog.com/2017/06/mobilenets-open-source-models-for.html
52. !52 Optimize for Mobile > IMAGE_SIZE=224 > ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}"
53. !53 Optimize for Mobile python -m tensorflow/examples/image_retraining/retrain.py \ --bottleneck_dir=tf_files/bottlenecks \ --how_many_training_steps=500 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --architecture="${ARCHITECTURE}" \ —image_dir=tf_files/fruit_photos
54. Version: November 2017 Device: Nexus 5X Average inference time: 1s
57. Model Inception V3 Quantized
58. Model MobileNets_1.0_224
59. Model MobileNets_0.5_224
60. Model MobileNets_0.25_224
61. Underneath the hood
62. Image Sampling Get Image from Camera Preview Crop the center square Resize Sample Image !62
63. Reference: https://jalammar.github.io/Supercharging-android-apps-using-tensorflow/ !63 Android SDK (Java) Android NDK (C++) Classifier Implementation TensorFlow JNI wrapper Image (Bitmap) Trained Model top_results Classifications + Confidence input_tensor 1 2 3 4 Camera Preview Overlay Display
64. Android FilesDir model.pb Labels model.pb label.txt
65. !65 May 2018 Annonce of ML Kit at Google I/O
66. ML Kit is in da house
67. TensorFlow Cloud Machine Learning Engine Ready to use Machine Learning API Use your own data to train models Cloud Vision API Cloud Speech API Cloud Translation API Cloud Natural Language API Cloud Video Intelligence API
68. What’s ML Kit for Firebase? !68 ML Kit Vision APIs ML Kit Custom Models
69. What’s ML Kit for Firebase? !69 Mobile Vision API TensorFlow Lite Android Neural Network API Google Cloud Vision API + + ML Kit Vision APIs ML Kit Custom Models
70. Natural Language Identify the language of text with ML Kit
71. Vision APIs
75. Source: https://firebase.google.com/docs/ml-kit/detect-faces !75 Face Contour Detection
79. Custom Model
80. TensorFlow lite model hosting On-device ML inference Automatic model fallback Over-the-air model updates !80
81. Reference: https://www.tensorflow.org/lite/convert/cmdline_examples !81 Convert model to TF Lite (model.tflite) Host your TF Lite model on Firebase Use the TF Lite model for inference Train your TF model (model.pb) GraphDef SavedModel tf.Keras model
82. !82 tflite_convert (post TF 1.9 float type) tflite_convert \ --graph_def_file=/tmp/magritte_retrained_graph.pb \ --output_file=/tmp/magritte_graph.tflite \ --inference_type=FLOAT \ --input_shape=1,224,224,3 \ --input_array=input \ --output_array=final_result \ --default_ranges_min=0 \ --default_ranges_max=6 *2 other input types: --saved_model_dir & --keras_model_file
83. !83 tflite_convert (post TF 1.9 float type) tflite_convert \ --graph_def_file=/tmp/magritte_retrained_graph.pb \ --output_file=/tmp/magritte_graph.tflite \ --inference_type=FLOAT \ --input_shape=1,224,224,3 \ --input_array=input \ --output_array=final_result \ --default_ranges_min=0 \ --default_ranges_max=6 *2 other input types: --saved_model_dir & --keras_model_file Reference: https://www.tensorflow.org/lite/convert/cmdline_reference
84. !84 tflite_convert (post TF 1.9 quantized type) tflite_convert \ --graph_def_file=/tmp/magritte_quantized_graph.pb \ --output_file=/tmp/magritte_graph.tflite \ --inference_type=QUANTIZED_UINT8 \ --input_shape=1,224,224,3 \ --input_array=input \ --output_array=final_result_fruits \ --default_ranges_min=0 \ --default_ranges_max=6 \ --mean_value=128 \ --std_value=128 \ * mean_vale & std_value are only needed if inference_input_type is QUANTIZED_UINT8. Reference: https://www.tensorflow.org/lite/convert/cmdline_reference
85. Obtain the TF Lite model !85 model.tflite label.txt apple banana grape kiwi orange pineapple strawberry watermelon lemon
87. FirebaseModelInterpreter FirebaseModelInputs FirebaseModelOutputs INPUT OUTPUT
88. !88 Input Dimensions & Constants companion object { private const val HOSTED_MODEL_NAME = "magritte" private const val LOCAL_MODEL_NAME = "magritte" private const val LOCAL_MODEL_PATH = "magritte.tflite" private const val LABEL_PATH = "magritte_labels.txt" const val DIM_BATCH_SIZE = 1 const val DIM_PIXEL_SIZE = 3 const val DIM_IMG_SIZE_X = 224 const val DIM_IMG_SIZE_Y = 224 private const val MEAN = 128 private const val STD = 128.0f }
89. !89 Set up custom model classifier val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build() val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
90. !90 Create local model source val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build() val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
91. !91 Create cloud model source val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build() val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
92. !92 Register model sources val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build() val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
93. !93 Set up FirebaseModelInterpreter val localModelSource = FirebaseLocalModelSource.Builder(LOCAL_MODEL_NAME) .setAssetFilePath(LOCAL_MODEL_ASSET).build() val cloudSource = FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() val manager = FirebaseModelManager.getInstance() manager.registerLocalModelSource(localModelSource) manager.registerCloudModelSource(cloudSource) val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName(LOCAL_MODEL_NAME) .build() interpreter = FirebaseModelInterpreter.getInstance(modelOptions)
94. !94 Input Dimensions // input & output options for non-quantized model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
95. !95 Output Dimensions // input & output options for non-quantized model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
96. !96 Input/Output Options // input & output options for non-quantized model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
97. !97 Input/Output Options // input & output options for non-quantized model val inputDims = intArrayOf(DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE) val outputDims = intArrayOf(1, labelList.size) inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.FLOAT32, inputDims) .setOutputFormat(0, FirebaseModelDataType.FLOAT32, outputDims) .build()
98. !98 Convert bitmap to ByteBuffer private val intValues = IntArray(DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y) private val imgData = ByteBuffer.allocateDirect( 4 * DIM_BATCH_SIZE * DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE) @Synchronized private fun convertBitmapToByteBuffer(bitmap: Bitmap): ByteBuffer { imgData.apply { order(ByteOrder.nativeOrder()) rewind() } bitmap.getPixels(intValues, 0, bitmap.width, 0, 0, bitmap.width, bitmap.height) // Preprocess the image data to normalized float intValues.forEach { imgData.putFloat(((it shr 16 and 0xFF) - MEAN) / STD) imgData.putFloat(((it shr 8 and 0xFF) - MEAN) / STD) imgData.putFloat(((it and 0xFF) - MEAN) / STD) } return imgData }
99. !99 Convert bitmap to ByteBuffer private val intValues = IntArray(DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y) private val imgData = ByteBuffer.allocateDirect( 4 * DIM_BATCH_SIZE * DIM_IMG_SIZE_X * DIM_IMG_SIZE_Y * DIM_PIXEL_SIZE) @Synchronized private fun convertBitmapToByteBuffer(bitmap: Bitmap): ByteBuffer { imgData.apply { order(ByteOrder.nativeOrder()) rewind() } bitmap.getPixels(intValues, 0, bitmap.width, 0, 0, bitmap.width, bitmap.height) // Preprocess the image data to normalized float intValues.forEach { imgData.putFloat(((it shr 16 and 0xFF) - MEAN) / STD) imgData.putFloat(((it shr 8 and 0xFF) - MEAN) / STD) imgData.putFloat(((it and 0xFF) - MEAN) / STD) } return imgData }
100. Normalized ByteBuffer FirebaseModelInputs INPUT
101. !101 Run the inference override fun process(bitmap: Bitmap) { val imageByteBuffer = convertBitmapToByteBuffer(bitmap) val inputs = FirebaseModelInputs.Builder().add(imageByteBuffer).build() interpreter?.run(inputs, inputOutputOptions) ?.addOnSuccessListener { val labelProbArray = it.getOutput<Array<FloatArray>>(0) val results = getTopLabel(labelProbArray) // … } } private fun getTopLabel(labelProbArray: Array<FloatArray>): Pair<String, Float> { return labelList.asSequence().mapIndexed { i, label -> Pair(label, labelProbArray[0][i]) }.sortedBy { it.second }.last() }
102. FirebaseModelOutputs OUTPUT apple banana grape kiwi orange pineapple strawberry watermelon lemon
103. Model MobileNets_1.0_224
104. Model MobileNets_1.0_224
105. Play Rock-Paper-Scissors-Spock-Lizard with your Android Things
107. Applications Node.js + Express Firebase (Hosting/Function/Storage) Android Things (Peripheral I/O APIs) Android SDK (TextToSpeech API/Camera API) TensorFlow Lite MobileNetV1 !107
108. Data Collector App !108
109. Collected Photos
110. PWM (Pulse Width Modulation) & Servo Motor !110
111. 16-Channel PWM Servo Driver !111
112. Android Things Starter Kit - NXP i.MX7D !112
113. !113
116. Source: !116
117. !117
118. Total Time Spent An entire weekend (of burning finger and slapped face)! My robot doesn't respect (yet) Asimov's Three Laws of Robotics !118
119. Take-away Image Credit: https://unsplash.com/photos/Kj2SaNHG-hg
120. TensorFlow & MobileNet TensorFlow Mobile (deprecated) . TensorFlow Lite / • Performance optimized for mobile and embedded devices • Tools support wider range of model formats conversion From MobileNetV1 to MobileNetV2 !120
121. Firebase ML Kit Still in Beta so API may break No callback for model downloading (yet) More features are coming Give feedbacks! !121
122. Other Trends Model Security How to protect your model? Cloud + Edge End-to-end AI infrastructure Machine Learning pipelines (e.g. KubeFlow) !122
123. Federate Learning Collaborative Machine Learning without Centralized Training Data !123 Source: https://research.googleblog.com/2017/04/federated-learning-collaborative.html
124. Google Translate for Sign Language !124
125. Thank you! ありがとうございました! Magritte: https://github.com/xebia-france/magritte ML Kit in Action: https://github.com/jinqian/MLKit-in-actions Hand Game Robot: https://www.hackster.io/bonbonking/ Twitter: @bonbonking
126. Resources • Artificial neural network: https://en.wikipedia.org/wiki/Artificial_neural_network • Deep Learning: https://en.wikipedia.org/wiki/Deep_learning • Convolutional Neural Network: https://en.wikipedia.org/wiki/Convolutional_neural_network • TensorFlow for Poets: https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/ • TensorFlow for Poets 2: Optimize for Mobile: https://codelabs.developers.google.com/codelabs/tensorflow- for-poets-2/ • TensorFlow Glossary: https://www.tensorflow.org/versions/r0.12/resources/glossary • Talk Magritte for DroidCon London: https://speakerdeck.com/jinqian/droidcon-london-heat-the-neurons- of-your-smartphone-with-deep-learning • Medium article: Android meets Machine Learning https://medium.com/xebia-france/android-meets- machine-learning-part-1-from-tensorflow-mobile-lite-to-ml-kit-4c7e6bc8eee3 !126