Push some Machine Learning into your App

English version of my previous talk, and up-to-date with new command line.
JokeBox gif: https://i.ibb.co/JrhHDKv/jokebox.gif
Corgi video: https://giant.gfycat.com/SardonicQuerulousAnemoneshrimp.webm
Monkey gif: https://i.ibb.co/RvJXZwq/monkey.gif
Blur gif: https://i.ibb.co/k4MJM2S/blur.gif

Transcript

1. Push Machine Learning into your apps … When TensorFlow and ML Kit rule the world
4. Machine Learning
5. Machine learning ? Supervised learning No Supervised learning Reinforcement learning Independent agent capable of learning from mistakes Clustering K-mean ... Decision Tree Logistic regression Boosting Neural Network …
6. Neural network R E S H A P E
7. a Neuron Linear operation Filter function input 1 input n output 1 output 1
8. a Neuron (with Pooling) input 1 input n output 1 output n POOLING Linear operation Filter function
9. Neural network R E S H A P E
11. TensorFlow High-performance numerical calculation tools Neural network via Deep Learning Two Mobile versions Open Source Made By Google Brain
13. Pre-trained Models
14. Inception V3 MobileNet Smart Reply ImageNet trained with trained with Accuracy ++ Size - Accuracy + Size ++
15. → Retrained MobileNet http://vision.stanford.edu/aditya86/ImageNetDogs/
16. Order the pictures corgi -----> cardigan -----> 0126.jpg 15431.jpg 6876784.jpg pembroke -----> cutedog.jpg pipou.jpg
17. RETRAIN https://www.tensorflow.org/hub/tutorials/image_retraining?hl=en tensorboard --logdir /tmp/retrain_logs
18. curl -LO https://github.com/tensorflow/hub/raw/master/examples/image_retraining/retrain.py RETRAIN https://www.tensorflow.org/hub/tutorials/image_retraining?hl=en
19. python retrain.py --image_dir corgi --output_labels quantops/labels.txt --output_graph quantops/graph.pb --tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/quantops/feature_vector/1 --how_many_training_steps 500 RETRAIN https://www.tensorflow.org/hub/tutorials/image_retraining?hl=en
20. TF model person label.txt
21. TensorBoard
22. TensorBoard
23. python retrain.py --image_dir corgi --output_labels float/labels.txt --output_graph float/graph.pb --tfhub_module https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/feature_vector/1 --how_many_training_steps 500 RETRAIN https://www.tensorflow.org/hub/tutorials/image_retraining?hl=en
24. TensorBoard
25. TensorBoard
26. TensorFlow Mobile
27. TensorFlow Mobile
28. Deprecated!
29. TensorFlow Lite
30. Lightweight solution Uses FlatBuffers models Optimized for mobile use Supports part of TensorFlow operations Still considered as a contribution to TensorFlow TensorFlow Lite ?
31. Optimisations : Quantization : FLOAT32 → BYTE8 Freeze : Cut branch, useless for prediction
32. T O C O TENSORFLOW LITE OPTIMIZING CONVERTER Saved Model, Frozen Graph or Keras → FlatBuffer
33. TOCO tflite_convert --output_file=quantops/graph.tflite --graph_def_file=quantops/graph.pb --inference_type=QUANTIZED_UINT8 --input_array Placeholder --output_arrays=final_result --mean_values=128 --std_dev_values=127
34. TOCO tflite_convert --output_file=float/graph.tflite --graph_def_file=float/graph.pb --inference_type=FLOAT --input_array Placeholder --output_arrays=final_result
35. Android
36. Integrate into Android : FlatBuffer Model + labels.txt Android Assets
37. Image → ByteBuffer private fun fromBitmapToByteBuffer(bitmap: Bitmap): ByteBuffer { val imgData = ByteBuffer.allocateDirect((if (isQuantized) 1 else 4) * IMG_SIZE * IMG_SIZE * 3) .apply { order(ByteOrder.nativeOrder()) rewind() } val pixels = IntArray(IMG_SIZE * IMG_SIZE) val createScaledBitmap = Bitmap.createScaledBitmap(bitmap, IMG_SIZE, IMG_SIZE, false) createScaledBitmap.apply { getPixels(pixels, 0, width, 0, 0, width, height) } pixels.forEach { value -> imgData.add(value shr 16 and 0xFF) imgData.add(value shr 8 and 0xFF) imgData.add(value and 0xFF) } return imgData } private fun ByteBuffer.add(pixel: Int) = if (isQuantized) put(pixel.toByte()) else putFloat((pixel - MEAN) / STD)
38. Interpreter val fileInputStream = context.assets.openFd(MODEL_NAME).let { FileInputStream(it.fileDescriptor).channel.map( FileChannel.MapMode.READ_ONLY, it.startOffset, it.declaredLength ) } val interpreter = Interpreter(fileInputStream) val labels = context.assets.open("labels.txt").bufferedReader().readLines()
39. Run ! fun recognizeDog(bitmap: Bitmap) { val imgData = fromBitmapToByteBuffer(bitmap) val outputs = Array(1, { FloatArray(labels.size) }) interpreter.run(imgData, outputs) val dog = labels .mapIndexed { index, label -> Pair(label, outputs[0][index]) } .sortedByDescending { it.second } .first() view?.displayDog(dog.first) }
40. ML KIT
41. ML Kit: Toolbox Mobile Vision + Google Cloud API + TensorFlow Lite
42. Based APIs
43. OCR (device & cloud) Faces detection (device) Barcode scanning (device) Image labelling (device & cloud) Landmark detection (cloud) Smart Reply
44. Example : Faces detection • Bounding polygon • Angles of rotation • Tracking ID • Facial landmarks ◦ Left eye ◦ Right eye ◦ Bottom of mouth ... • Feature probabilities ◦ Smiling ◦ Left eye open ◦ Right eye open • Facial contours: ◦ Nose bridge ◦ Left eye ◦ Top of upper lip
45. Example : Faces detection • Bounding polygon • Angles of rotation • Tracking ID • Facial landmarks ◦ Left eye ◦ Right eye ◦ Bottom of mouth ... • Feature probabilities ◦ Smiling ◦ Left eye open ◦ Right eye open • Facial contours: ◦ Nose bridge ◦ Left eye ◦ Top of upper lip...
46. Example : Faces detection The Joke Box If you don't smile, it offers you a joke If you smile, everything is fine. API : Dad Jokes https://icanhazdadjoke.com/ Confettis : nl.dionsegijn:konfetti Camera (don’t talk about Camera2) : com.camerakit:camerakit (Warning: beautiful design) In collaboration with Nicolas Telera
47. Example : Faces detection implementation 'com.google.firebase:firebase-ml-vision:17.0.1' implementation 'com.google.firebase:firebase-core:16.0.4' [...] apply plugin: 'com.google.gms.google-services' Don't forget to register your app on Firebase and integrate google.services
48. Example : Faces detection init { val options = FirebaseVisionFaceDetectorOptions .Builder() .setClassificationType( FirebaseVisionFaceDetectorOptions .ALL_CLASSIFICATIONS ) .build() detector = FirebaseVision.getInstance().getVisionFaceDetector(options) }
49. fun recognizePicture(bitmap: Bitmap) { } Example : Faces detection val firebaseVisionImage = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { faces -> } .addOnFailureListener { view.displayFail() } try { if (faces.first().smilingProbability > 0.70) { view.displaySmile() } else { view.displaySad() } } catch (e: NoSuchElementException) { view.displayFail() }
51. Device or Cloud ? Performance Device < Performance Cloud Cloud pricing: “Free for first 1000 uses of this feature per month” Device: possible Offline! Warning: data protection
52. CUSTOM MODEL with TensorFlow Lite
53. ML Kit Custom Android + iOS
54. ML Kit Custom Android + iOS
55. ML Kit Custom Android + iOS
56. ML Kit Custom Android + iOS
57. Model : - Local (Asset) - Cloud - Both !
58. Add the template and label file corgi.tflite + labels.txt Android Assets
59. Dependencies implementation 'com.google.firebase:firebase-ml-model-interpreter:17.0.3' implementation 'com.google.firebase:firebase-core:16.0.7' [...] apply plugin: 'com.google.gms.google-services' Don't forget to register your app on Firebase and integrate google.services
60. Initialization val dataType = if (isQuantized) FirebaseModelDataType.BYTE else FirebaseModelDataType.FLOAT32 dataOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, dataType, intArrayOf(1, IMG_SIZE, IMG_SIZE, 3)) .setOutputFormat(0, dataType, intArrayOf(1, labels.size)) .build() val labels = context.assets.open("labels.txt").bufferedReader().readLines()
61. Initialization https://lutzroeder.github.io/netron/
62. Interpreter initialization: Local Source val localSource = FirebaseLocalModelSource .Builder(ASSET) .setAssetFilePath("$MODEL_NAME.tflite") .build()
63. Interpreter initialization: Cloud Source val conditions = FirebaseModelDownloadConditions .Builder() .requireWifi() .build() val cloudSource = FirebaseCloudModelSource.Builder(MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build()
64. Interpreter initialization: Cloud Source val conditions = FirebaseModelDownloadConditions .Builder() .requireWifi() .build() val cloudSource = FirebaseCloudModelSource.Builder(MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() Other conditions >= Nougat: requireCharging(), requireDeviceIdle()
65. Interpreter initialization: Cloud Source val conditions = FirebaseModelDownloadConditions .Builder() .requireWifi() .build() val cloudSource = FirebaseCloudModelSource.Builder(MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build()
66. Interpreter initialization FirebaseModelManager.getInstance().apply { registerLocalModelSource(localSource) registerCloudModelSource(cloudSource) } val interpreter = FirebaseModelInterpreter.getInstance( FirebaseModelOptions.Builder() .setCloudModelName(MODEL_NAME) .setLocalModelName(ASSET) .build() )
67. Run ! val inputs = FirebaseModelInputs.Builder() .add(fromBitmapToByteBuffer(bitmap)) .build() interpreter?.run(inputs, dataOptions) ?.addOnSuccessListener { val output = it.getOutput<Array<FloatArray>>(0) val label = labels .mapIndexed { index, label ->Pair(label, output[0][index])} .maxBy { it.second }.first() view?.displayDog(label) } ?.addOnFailureListener { view?.displayError() }
70. But : Download the model is long and random What about the bugs in TensorFlow Lite? TOCO, quantized model and other misunderstandings Incomplete documentation Difficult examples (with quite dirty code) Online Based API pricing
71. Usages
72. Lose It Nutrition facts with ML Kit Text Recognition API https://developers.google.com/ml-kit/ “Snap it” features to recognize food with TensorFlow Lite https://loseitblog.com/2017/11/10/on-device-image-recognition/
73. Anonymize By Nicolas Telera Detect and blur faces in a picture. https://github.com/NicolasTelera/Anonymize
74. Why not in your app? Fraud detection User help Adding documents Games Picture filter Automatic photo tags Proposal for hashtags and your ideas?
75. References : https://firebase.google.com/docs/ml-kit/ https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ https://codelabs.developers.google.com/codelabs/mlkit-android/ Dataset : https://www.kaggle.com/slothkong/10-monkey-species/version/1 @SandraDdev /SandraJam @sandra.dupre