Who's afraid of ML? V2 : First steps with MlKit

Who's afraid of ML? V2 : First steps with MlKit

Although it is buzzing nowadays, machine learning is still a mystery to many of us.
However, worry not! Making your app smarter has just become possible with the new Firebase ML-Kit.
On this talk, we’ll get to know some basic ML concepts to clear the air around the topic: what Machine Learning even means? how does it work? what does it have to do with our actual brain? Then, we'll explore how to create our first machine learning Android app with ML-Kit and TensorFlow Lite!

Blog post series: http://bit.ly/brittML-P1

3142db3adb711e247e371153b5777e04?s=128

Britt Barak

August 20, 2018
Tweet

Transcript

  1. Who is afraid of Machine Learning? Britt Barak & First

    Steps With ML-Kit V.2 @Britt Barak
  2. Britt Barak DevRel - Nexmo Google Developer Expert Women Techmakers

    Israel Britt Barak @brittBarak
  3. @BrittBarak

  4. @BrittBarak

  5. @BrittBarak

  6. @BrittBarak

  7. @BrittBarak

  8. = @BrittBarak

  9. @BrittBarak

  10. @BrittBarak

  11. . @BrittBarak

  12. neXmo

  13. Strawberry Not Strawberry @BrittBarak

  14. This is a strawberry @BrittBarak

  15. This is a strawberry Red Seeds pattern Top leaves @BrittBarak

  16. Input Red Seeds pattern Top leaves @BrittBarak

  17. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves @BrittBarak

  18. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves @BrittBarak

  19. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves @BrittBarak

  20. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 @BrittBarak
  21. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 @BrittBarak
  22. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 @BrittBarak
  23. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 @BrittBarak
  24. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  25. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 1.74

    0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  26. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 1.74

    0.5 0.8 0.3 @BrittBarak
  27. 0.64 0.75 0.4 Input Red Seeds pattern 1.74 Top leaves

    @BrittBarak
  28. 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97

    Top leaves @BrittBarak
  29. 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97

    Top leaves Output Not Strawberry Strawberry @BrittBarak
  30. Not Strawberry Strawberry 0.64 0.75 0.4 Input Red Seeds pattern

    1.02 1.74 0.97 Top leaves Output @BrittBarak
  31. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  32. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  33. @BrittBarak

  34. Not Strawberry Strawberry 0.7 0.03 0.01 Input Red Seeds pattern

    Top leaves Output @BrittBarak
  35. Not Strawberry Strawberry 0.7 0.03 0.01 Input Red Seeds pattern

    3.72 0.89 1.92 Top leaves Output @BrittBarak
  36. Not Strawberry Strawberry 0.2 0.8 0.7 0.03 0.01 Input Red

    Seeds pattern 3.72 0.89 1.92 Top leaves Output @BrittBarak
  37. Strawberry Not Not Not Strawberry Strawberry Not Not Not 0.5

    * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  38. Training @BrittBarak

  39. None
  40. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  41. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern Top leaves Output Hidden @BrittBarak
  42. Data science

  43. Pre Trained Model

  44. Strawberry @BrittBarak

  45. Strawberry @BrittBarak

  46. Why a local model? •Security •Privacy •Latency •Bandwith •Performance •Offline

    support •… @BrittBarak
  47. beta

  48. •Out-of-the-box use cases •On-device / cloud management •Deploy custom models

    beta @BrittBarak
  49. Text Recognition @BrittBarak

  50. Face Detection @BrittBarak

  51. Barcode Scanning @BrittBarak

  52. Landmark Recognition @BrittBarak

  53. Image Labeling @BrittBarak

  54. Custom Models @BrittBarak

  55. Example

  56. Image Classifier classifier .classify(bitmap) label @BrittBarak

  57. •Run on device model •Run on cloud model •Run a

    custom model Image Classifier Device Model Cloud Model Custom Model @BrittBarak
  58. Image Classifier Device Model Cloud Model Custom Model @BrittBarak

  59. 1. Setup Firebase Detector @BrittBarak

  60. FirebaseVision.getInstance() @BrittBarak

  61. Image Classifier FirebaseVisionLabelDetector localDetector = FirebaseVision.getInstance() .getVisionLabelDetector() @BrittBarak

  62. 2. Process input @BrittBarak

  63. FirebaseVisionImage •Bitmap •byteArray •byteBuffer •image Uri •Media Image @BrittBarak

  64. Image Classifier image = FirebaseVisionImage.fromBitmap(bitmap) @BrittBarak

  65. 3. Execute the model @BrittBarak

  66. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels,

    callback); } }) } @BrittBarak
  67. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels,

    callback); } }) @BrittBarak
  68. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels,

    callback); } }) @BrittBarak
  69. 4. Process result @BrittBarak

  70. Image Classifier localDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>(){ void onSuccess(List<FirebaseVisionLabel> labels){ processResult(labels,

    callback); } }) @BrittBarak
  71. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  72. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  73. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  74. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  75. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  76. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  77. Result : @BrittBarak

  78. Result : @BrittBarak

  79. Result : @BrittBarak

  80. Image Classifier Device Model Cloud Model Custom Model @BrittBarak

  81. 1. Setup Firebase Detector @BrittBarak

  82. Image Classifier cloudDetector = FirebaseVision.getInstance() .getVisionCloudLabelDetector() @BrittBarak

  83. 2. Process input [the same] @BrittBarak

  84. Image Classifier image = FirebaseVisionImage.fromBitmap(bitmap) @BrittBarak

  85. 3. Execute the model [same - almost!] @BrittBarak

  86. Image Classifier cloudDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionCloudLabel>>(){ void onSuccess(List<FirebaseVisionCloudLabel>labels){ processResult(labels, callback);

    } }) @BrittBarak
  87. Image Classifier cloudDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionCloudLabel>>(){ void onSuccess(List<FirebaseVisionCloudLabel>labels){ processResult(labels, callback);

    } }) @BrittBarak
  88. Image Classifier cloudDetector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionCloudLabel>>(){ void onSuccess(List<FirebaseVisionCloudLabel>labels){ processResult(labels, callback);

    } }) @BrittBarak
  89. 4. Process result [same - almost!] @BrittBarak

  90. Result : @BrittBarak

  91. Result : @BrittBarak

  92. Result : @BrittBarak

  93. Result : @BrittBarak

  94. Result : @BrittBarak

  95. Result : @BrittBarak

  96. Result : @BrittBarak

  97. Result : @BrittBarak

  98. Image Classifier Device Model Cloud Model Custom Model @BrittBarak

  99. TensorFlow - Open source - Widely used - Flexible for

    scale: - 1 or more CPUs / GPUs - desktop, server, mobile device @BrittBarak
  100. TensorFlow Mobile - Speech Recognition - Image Recognition - Object

    Localization - Gesture Recognition - Translation - Text Classification - Voice Synthesis @BrittBarak
  101. Lightweight Fast Cross platform @BrittBarak

  102. MobileNet Inception -V3 SmartReply Models @BrittBarak

  103. beta @BrittBarak

  104. Image Classifier classifier .classify(bitmap) label @BrittBarak

  105. 0. Host the model @BrittBarak

  106. Where to host? •Remotely •On device @BrittBarak

  107. 0.1. Host on Firebase @BrittBarak

  108. 0.2 Save on device @BrittBarak

  109. 0.2 Save on device build.gradle android {
 aaptOptions {
 noCompress

    "tflite"
 }
 } @BrittBarak
  110. 0.1. Load the model @BrittBarak

  111. Image Classifier void configureModelDownloadManager() { FirebaseModelManager modelManager = FirebaseModelManager.getInstance(); }

    @BrittBarak
  112. 0.1.1 config local model @BrittBarak

  113. Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource

    .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
  114. Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource

    .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
  115. 0.1.2 config hosted model @BrittBarak

  116. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  117. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  118. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  119. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  120. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  121. Image Classifier void configureModelDownloadManager() { //… conditions = new FirebaseModelDownloadConditions

    .Builder() .requireWifi() .build(); } @BrittBarak
  122. Image Classifier void configureModelDownloadManager() { //… conditions = new FirebaseModelDownloadConditions

    .Builder() .requireWifi() .build(); } @BrittBarak
  123. 1. Setup the interpreter @BrittBarak

  124. classifier .classify(bitmap Image Classifier Labels label Input ByteBuffer Output ProbArray

    Model
  125. Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[]

    outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
  126. Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[]

    outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
  127. 0.2 Save on device @BrittBarak

  128. @BrittBarak

  129. Image Classifier //… mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims)

    .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
  130. Image Classifier //… modelOptions = new FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName("asset") .build();

    interpreter = FirebaseModelInterpreter .getInstance(modelOptions); @BrittBarak
  131. Image Classifier //… modelOptions = new FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL_NAME) .setLocalModelName("asset") .build();

    interpreter = FirebaseModelInterpreter .getInstance(modelOptions); @BrittBarak
  132. 1. Process Input @BrittBarak

  133. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs, start); } @BrittBarak
  134. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs,); } @BrittBarak
  135. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
  136. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
  137. 2. Execute Model @BrittBarak

  138. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  139. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  140. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  141. 3. Process output @BrittBarak

  142. probArray = { [0.7], [0.3], [0], [0], } 0.3 labelList

    = { apple, strawberry, banana, pineapple, } @BrittBarak
  143. Result : @BrittBarak

  144. Result : @BrittBarak

  145. Result : @BrittBarak

  146. Result : @BrittBarak

  147. Result : @BrittBarak

  148. A new dawn of Machine Learning @BrittBarak

  149. Links - Tensorflow - https://www.tensorflow.org/ - Tensorflow lite - https://www.tensorflow.org/mobile/tflite/

    - Codes labs - codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ - Google’s Machine Learning Crash Course - developers.google.com/machine-learning/crash-course/ - [Dr. Joe Dispenza] @BrittBarak
  150. Thank you! Keep in touch! Britt Barak @brittBarak