Who's afraid of ML? V2 : First steps with MlKit

Who's afraid of ML? V2 : First steps with MlKit

Although it is buzzing nowadays, machine learning is still a mystery to many of us.
However, worry not! Making your app smarter has just become possible with the new Firebase ML-Kit.
On this talk, we’ll get to know some basic ML concepts to clear the air around the topic: what Machine Learning even means? how does it work? what does it have to do with our actual brain? Then, we'll explore how to create our first machine learning Android app with ML-Kit and TensorFlow Lite!

Blog post series: http://bit.ly/brittML-P1

3142db3adb711e247e371153b5777e04?s=128

Britt Barak

August 20, 2018
Tweet

Transcript

  1. 1.

    Who is afraid of Machine Learning? Britt Barak & First

    Steps With ML-Kit V.2 @Britt Barak
  2. 12.
  3. 21.

    0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 @BrittBarak
  4. 22.

    0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 @BrittBarak
  5. 23.

    0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 @BrittBarak
  6. 24.

    0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  7. 25.

    0.64 0.75 0.4 Input Red Seeds pattern Top leaves 1.74

    0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  8. 29.

    0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97

    Top leaves Output Not Strawberry Strawberry @BrittBarak
  9. 30.

    Not Strawberry Strawberry 0.64 0.75 0.4 Input Red Seeds pattern

    1.02 1.74 0.97 Top leaves Output @BrittBarak
  10. 31.

    Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  11. 32.

    Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  12. 35.

    Not Strawberry Strawberry 0.7 0.03 0.01 Input Red Seeds pattern

    3.72 0.89 1.92 Top leaves Output @BrittBarak
  13. 36.

    Not Strawberry Strawberry 0.2 0.8 0.7 0.03 0.01 Input Red

    Seeds pattern 3.72 0.89 1.92 Top leaves Output @BrittBarak
  14. 37.

    Strawberry Not Not Not Strawberry Strawberry Not Not Not 0.5

    * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  15. 39.
  16. 40.

    Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  17. 41.

    Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern Top leaves Output Hidden @BrittBarak
  18. 47.
  19. 55.
  20. 57.

    •Run on device model •Run on cloud model •Run a

    custom model Image Classifier Device Model Cloud Model Custom Model @BrittBarak
  21. 71.

    Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  22. 72.

    Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  23. 73.

    Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  24. 74.

    Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  25. 75.

    Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  26. 76.

    Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  27. 99.

    TensorFlow - Open source - Widely used - Flexible for

    scale: - 1 or more CPUs / GPUs - desktop, server, mobile device @BrittBarak
  28. 100.

    TensorFlow Mobile - Speech Recognition - Image Recognition - Object

    Localization - Gesture Recognition - Translation - Text Classification - Voice Synthesis @BrittBarak
  29. 113.

    Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource

    .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
  30. 114.

    Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource

    .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
  31. 116.

    Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  32. 117.

    Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  33. 118.

    Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  34. 119.

    Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  35. 120.

    Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  36. 125.

    Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[]

    outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
  37. 126.

    Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[]

    outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
  38. 133.

    Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs, start); } @BrittBarak
  39. 134.

    Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs,); } @BrittBarak
  40. 135.

    Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
  41. 136.

    Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
  42. 138.

    Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  43. 139.

    Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  44. 140.

    Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  45. 142.

    probArray = { [0.7], [0.3], [0], [0], } 0.3 labelList

    = { apple, strawberry, banana, pineapple, } @BrittBarak
  46. 149.

    Links - Tensorflow - https://www.tensorflow.org/ - Tensorflow lite - https://www.tensorflow.org/mobile/tflite/

    - Codes labs - codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ - Google’s Machine Learning Crash Course - developers.google.com/machine-learning/crash-course/ - [Dr. Joe Dispenza] @BrittBarak