Who's afraid of ML? V2 : First steps with MlKit

Although it is buzzing nowadays, machine learning is still a mystery to many of us.
However, worry not! Making your app smarter has just become possible with the new Firebase ML-Kit.
On this talk, we’ll get to know some basic ML concepts to clear the air around the topic: what Machine Learning even means? how does it work? what does it have to do with our actual brain? Then, we'll explore how to create our first machine learning Android app with ML-Kit and TensorFlow Lite!

Blog post series: http://bit.ly/brittML-P1

Transcript

1. Who is afraid of Machine Learning? Britt Barak & First Steps With ML-Kit V.2 @Britt Barak
2. Britt Barak DevRel - Nexmo Google Developer Expert Women Techmakers Israel Britt Barak @brittBarak
3. @BrittBarak
4. @BrittBarak
5. @BrittBarak
6. @BrittBarak
7. @BrittBarak
8. = @BrittBarak
9. @BrittBarak
10. @BrittBarak
11. . @BrittBarak
12. neXmo
13. Strawberry Not Strawberry @BrittBarak
14. This is a strawberry @BrittBarak
15. This is a strawberry Red Seeds pattern Top leaves @BrittBarak
16. Input Red Seeds pattern Top leaves @BrittBarak
17. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves @BrittBarak
18. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves @BrittBarak
19. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves @BrittBarak
20. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5 0.8 0.3 @BrittBarak
21. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 @BrittBarak
22. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 @BrittBarak
23. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 @BrittBarak
24. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
25. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 1.74 0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
26. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 1.74 0.5 0.8 0.3 @BrittBarak
27. 0.64 0.75 0.4 Input Red Seeds pattern 1.74 Top leaves @BrittBarak
28. 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97 Top leaves @BrittBarak
29. 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97 Top leaves Output Not Strawberry Strawberry @BrittBarak
30. Not Strawberry Strawberry 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
31. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
32. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
33. @BrittBarak
34. Not Strawberry Strawberry 0.7 0.03 0.01 Input Red Seeds pattern Top leaves Output @BrittBarak
35. Not Strawberry Strawberry 0.7 0.03 0.01 Input Red Seeds pattern 3.72 0.89 1.92 Top leaves Output @BrittBarak
36. Not Strawberry Strawberry 0.2 0.8 0.7 0.03 0.01 Input Red Seeds pattern 3.72 0.89 1.92 Top leaves Output @BrittBarak
37. Strawberry Not Not Not Strawberry Strawberry Not Not Not 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
38. Training @BrittBarak
40. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
41. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red Seeds pattern Top leaves Output Hidden @BrittBarak
42. Data science
43. Pre Trained Model
44. Strawberry @BrittBarak
45. Strawberry @BrittBarak
46. Why a local model? •Security •Privacy •Latency •Bandwith •Performance •Offline support •… @BrittBarak
47. beta
48. •Out-of-the-box use cases •On-device / cloud management •Deploy custom models beta @BrittBarak
49. Text Recognition @BrittBarak
50. Face Detection @BrittBarak
51. Barcode Scanning @BrittBarak
52. Landmark Recognition @BrittBarak
53. Image Labeling @BrittBarak
54. Custom Models @BrittBarak
55. Example
56. Image Classifier classifier .classify(bitmap) label @BrittBarak
57. •Run on device model •Run on cloud model •Run a custom model Image Classifier Device Model Cloud Model Custom Model @BrittBarak
58. Image Classifier Device Model Cloud Model Custom Model @BrittBarak
59. 1. Setup Firebase Detector @BrittBarak
60. FirebaseVision.getInstance() @BrittBarak
61. Image Classifier FirebaseVisionLabelDetector localDetector = FirebaseVision.getInstance() .getVisionLabelDetector() @BrittBarak
62. 2. Process input @BrittBarak
63. FirebaseVisionImage •Bitmap •byteArray •byteBuffer •image Uri •Media Image @BrittBarak
64. Image Classifier image = FirebaseVisionImage.fromBitmap(bitmap) @BrittBarak
65. 3. Execute the model @BrittBarak
66. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels, callback); } }) } @BrittBarak
67. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels, callback); } }) @BrittBarak
68. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels, callback); } }) @BrittBarak
69. 4. Process result @BrittBarak
70. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels, callback); } }) @BrittBarak
71. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator); for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
72. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator); for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
73. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator); for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
74. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator); for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
75. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator); for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
76. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator); for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
77. Result : @BrittBarak
78. Result : @BrittBarak
79. Result : @BrittBarak
80. Image Classifier Device Model Cloud Model Custom Model @BrittBarak
81. 1. Setup Firebase Detector @BrittBarak
82. Image Classifier cloudDetector = FirebaseVision.getInstance() .getVisionCloudLabelDetector() @BrittBarak
83. 2. Process input [the same] @BrittBarak
84. Image Classifier image = FirebaseVisionImage.fromBitmap(bitmap) @BrittBarak
85. 3. Execute the model [same - almost!] @BrittBarak
86. Image Classifier cloudDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionCloudLabel>>(){ void onSuccess(List<FirebaseVisionCloudLabel>labels){ processResult(labels, callback); } }) @BrittBarak
87. Image Classifier cloudDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionCloudLabel>>(){ void onSuccess(List<FirebaseVisionCloudLabel>labels){ processResult(labels, callback); } }) @BrittBarak
88. Image Classifier cloudDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionCloudLabel>>(){ void onSuccess(List<FirebaseVisionCloudLabel>labels){ processResult(labels, callback); } }) @BrittBarak
89. 4. Process result [same - almost!] @BrittBarak
90. Result : @BrittBarak
91. Result : @BrittBarak
92. Result : @BrittBarak
93. Result : @BrittBarak
94. Result : @BrittBarak
95. Result : @BrittBarak
96. Result : @BrittBarak
97. Result : @BrittBarak
98. Image Classifier Device Model Cloud Model Custom Model @BrittBarak
99. TensorFlow - Open source - Widely used - Flexible for scale: - 1 or more CPUs / GPUs - desktop, server, mobile device @BrittBarak
100. TensorFlow Mobile - Speech Recognition - Image Recognition - Object Localization - Gesture Recognition - Translation - Text Classification - Voice Synthesis @BrittBarak
101. Lightweight Fast Cross platform @BrittBarak
102. MobileNet Inception -V3 SmartReply Models @BrittBarak
103. beta @BrittBarak
104. Image Classifier classifier .classify(bitmap) label @BrittBarak
105. 0. Host the model @BrittBarak
106. Where to host? •Remotely •On device @BrittBarak
107. 0.1. Host on Firebase @BrittBarak
108. 0.2 Save on device @BrittBarak
109. 0.2 Save on device build.gradle android {
 aaptOptions {
 noCompress "tflite"
 }
 } @BrittBarak
110. 0.1. Load the model @BrittBarak
111. Image Classifier void configureModelDownloadManager() { FirebaseModelManager modelManager = FirebaseModelManager.getInstance(); } @BrittBarak
112. 0.1.1 config local model @BrittBarak
113. Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
114. Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
115. 0.1.2 config hosted model @BrittBarak
116. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
117. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
118. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
119. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
120. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
121. Image Classifier void configureModelDownloadManager() { //… conditions = new FirebaseModelDownloadConditions .Builder() .requireWifi() .build(); } @BrittBarak
122. Image Classifier void configureModelDownloadManager() { //… conditions = new FirebaseModelDownloadConditions .Builder() .requireWifi() .build(); } @BrittBarak
123. 1. Setup the interpreter @BrittBarak
124. classifier .classify(bitmap Image Classifier Labels label Input ByteBuffer Output ProbArray Model
125. Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[] outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
126. Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[] outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
127. 0.2 Save on device @BrittBarak
128. @BrittBarak
129. Image Classifier //… mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
130. Image Classifier //… modelOptions = new FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName("asset") .build(); interpreter = FirebaseModelInterpreter .getInstance(modelOptions); @BrittBarak
131. Image Classifier //… modelOptions = new FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName("asset") .build(); interpreter = FirebaseModelInterpreter .getInstance(modelOptions); @BrittBarak
132. 1. Process Input @BrittBarak
133. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage); inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs, start); } @BrittBarak
134. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage); inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs,); } @BrittBarak
135. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage); inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
136. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage); inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
137. 2. Execute Model @BrittBarak
138. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){ @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
139. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){ @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
140. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){ @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
141. 3. Process output @BrittBarak
142. probArray = { [0.7], [0.3], [0], [0], } 0.3 labelList = { apple, strawberry, banana, pineapple, } @BrittBarak
143. Result : @BrittBarak
144. Result : @BrittBarak
145. Result : @BrittBarak
146. Result : @BrittBarak
147. Result : @BrittBarak
148. A new dawn of Machine Learning @BrittBarak
149. Links - Tensorflow - https://www.tensorflow.org/ - Tensorflow lite - https://www.tensorflow.org/mobile/tflite/ - Codes labs - codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ - Google’s Machine Learning Crash Course - developers.google.com/machine-learning/crash-course/ - [Dr. Joe Dispenza] @BrittBarak
150. Thank you! Keep in touch! Britt Barak @brittBarak