$30 off During Our Annual Pro Sale. View Details »

Who's afraid of ML? V2 : First steps with MlKit

Who's afraid of ML? V2 : First steps with MlKit

Although it is buzzing nowadays, machine learning is still a mystery to many of us.
However, worry not! Making your app smarter has just become possible with the new Firebase ML-Kit.
On this talk, we’ll get to know some basic ML concepts to clear the air around the topic: what Machine Learning even means? how does it work? what does it have to do with our actual brain? Then, we'll explore how to create our first machine learning Android app with ML-Kit and TensorFlow Lite!

Blog post series: http://bit.ly/brittML-P1

Britt Barak

August 20, 2018
Tweet

More Decks by Britt Barak

Other Decks in Technology

Transcript

  1. Who is afraid of Machine Learning? Britt Barak & First

    Steps With ML-Kit V.2 @Britt Barak
  2. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 @BrittBarak
  3. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 @BrittBarak
  4. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 @BrittBarak
  5. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 0.5

    0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  6. 0.64 0.75 0.4 Input Red Seeds pattern Top leaves 1.74

    0.5 0.8 0.3 0.5 * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  7. 0.64 0.75 0.4 Input Red Seeds pattern 1.02 1.74 0.97

    Top leaves Output Not Strawberry Strawberry @BrittBarak
  8. Not Strawberry Strawberry 0.64 0.75 0.4 Input Red Seeds pattern

    1.02 1.74 0.97 Top leaves Output @BrittBarak
  9. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  10. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  11. Not Strawberry Strawberry 0.7 0.03 0.01 Input Red Seeds pattern

    3.72 0.89 1.92 Top leaves Output @BrittBarak
  12. Not Strawberry Strawberry 0.2 0.8 0.7 0.03 0.01 Input Red

    Seeds pattern 3.72 0.89 1.92 Top leaves Output @BrittBarak
  13. Strawberry Not Not Not Strawberry Strawberry Not Not Not 0.5

    * 0.64 + 0.8 * 0.75 + 0.3 * 0.4 1.04 + 0.7 1.74 @BrittBarak
  14. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern 1.02 1.74 0.97 Top leaves Output @BrittBarak
  15. Not Strawberry Strawberry 0.87 0.13 0.64 0.75 0.4 Input Red

    Seeds pattern Top leaves Output Hidden @BrittBarak
  16. •Run on device model •Run on cloud model •Run a

    custom model Image Classifier Device Model Cloud Model Custom Model @BrittBarak
  17. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  18. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  19. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  20. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  21. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  22. Image Classifier void processResult(List<FirebaseVisionLabel> labels, ClassifierCallback callback) { resultLabels.clear(); labels.sort(localLabelComparator);

    for (int i = 0; i < Math.min(RESULTS_TO_SHOW,labels.size()); ++i) { FirebaseVisionLabel label = labels.get(i); resultLabels.add( label.getLabel() + ":" + label.getConfidence()); } callback.onClassified(resultLabels); } @BrittBarak
  23. TensorFlow - Open source - Widely used - Flexible for

    scale: - 1 or more CPUs / GPUs - desktop, server, mobile device @BrittBarak
  24. TensorFlow Mobile - Speech Recognition - Image Recognition - Object

    Localization - Gesture Recognition - Translation - Text Classification - Voice Synthesis @BrittBarak
  25. Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource

    .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
  26. Image Classifier void configureModelDownloadManager() { //… localModelSource = new FirebaseLocalModelSource

    .Builder("asset") .setAssetFilePath(LOCAL_MODEL_ASSET) .build(); modelManager. registerLocalModelSource(localModelSource); } @BrittBarak
  27. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  28. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  29. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  30. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  31. Image Classifier void configureModelDownloadManager() { //… cloudSource = new FirebaseCloudModelSource.Builder(HOSTED_MODEL_NAME)

    .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); modelManager.registerCloudModelSource(cloudSource); } @BrittBarak
  32. Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[]

    outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
  33. Image Classifier int[] inputDims = {DIM_BATCH_SIZE, DIM_IMG_SIZE_X, DIM_IMG_SIZE_Y, DIM_PIXEL_SIZE}; int[]

    outputDims = {DIM_BATCH_SIZE, mLabelList.size()}; mDataOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, inputDims) .setOutputFormat(0, FirebaseModelDataType.BYTE, outputDims) .build(); @BrittBarak
  34. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs, start); } @BrittBarak
  35. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs,); } @BrittBarak
  36. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
  37. Image Classifier void executeCustom(Bitmap selectedImage) { //… imgData = convertBitmapToByteBuffer(selectedImage);

    inputs = new FirebaseModelInputs .Builder() .add(imgData) .build(); runCustomModel(inputs); } @BrittBarak
  38. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  39. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  40. Image Classifier void runCustomModel(…){ interpreter .run(inputs, mDataOptions) .continueWith(new Continuation<FirebaseModelOutputs, List<String>>(){

    @Override List<String>then (Task<FirebaseModelOutputs> task){ return processResult(task); } }); } @BrittBarak
  41. probArray = { [0.7], [0.3], [0], [0], } 0.3 labelList

    = { apple, strawberry, banana, pineapple, } @BrittBarak
  42. Links - Tensorflow - https://www.tensorflow.org/ - Tensorflow lite - https://www.tensorflow.org/mobile/tflite/

    - Codes labs - codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ - Google’s Machine Learning Crash Course - developers.google.com/machine-learning/crash-course/ - [Dr. Joe Dispenza] @BrittBarak