Android Developers Guide to Machine Learning with MLkit and TensorFlow

Android Developers Guide to Machine Learning with MLkit and TensorFlow

Presented at the Google Developer Agency Day Johannesburg 2018.

Spoke about MLKit, Firebase and Custom TensorFlow models.

2a37bf1e025cc1523124774c760df91a?s=128

Rebecca Franks

June 28, 2018
Tweet

Transcript

  1. Android Developers Guide to Machine Learning With MLKit, Tensorflow &

    Firebase
  2. Rebecca Franks @riggaroo Google Developer Expert Android @ Over Pluralsight

    Author GDG Johannesburg Organiser
  3. Who considers themselves an expert in Machine Learning?

  4. None
  5. What is Machine Learning?

  6. Machine learning is an application of Artificial Intelligence in which

    we input a lot of data and let the machines learn “by themselves”
  7. None
  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None
  15. Face detection$% W orks offline

  16. None
  17. None
  18. None
  19. None
  20. None
  21. None
  22. Demo

  23. None
  24. ⛩ Landmark detection

  25. None
  26. None
  27. Demo

  28. None
  29. Image Labelling W orks offline

  30. None
  31. Barcode scanning W orks offline

  32. None
  33. None
  34. val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_QR_CODE) .build() val image = FirebaseVisionImage.fromBitmap(bitmap)

    val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it))
  35. val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_QR_CODE) .build() val image = FirebaseVisionImage.fromBitmap(bitmap)

    val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image) .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result)
  36. val image = FirebaseVisionImage.fromBitmap(bitmap) val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) detector.detectInImage(image)

    .addOnSuccessListener { processedBitmap.postValue(barcodeProcessor.drawBoxes(bitmap, it)) var result = String() it.forEach { result += "VALUE TYPE: ${it.valueType} Raw Value: ${it.rawValue}" textResult.postValue(result) } }.addOnFailureListener{ textResult.postValue(it.message) }
  37. None
  38. OCR W orks offline

  39. None
  40. On Device vs Cloud

  41. private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val

    firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result)
  42. private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val

    firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) } } .addOnFailureListener{ Toast.makeText(/../“Error detecting Text $it”/../) }
  43. private fun doOcrDetection(bitmap: Bitmap){ val detector = FirebaseVision.getInstance() .visionTextDetector val

    firebaseImage = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(firebaseImage) .addOnSuccessListener { processedBitmap.postValue(ocrProcessor.drawBoxes(bitmap, it)) var result = String() it.blocks.forEach { result += " " + it.text textResult.postValue(result) } } .addOnFailureListener{ Toast.makeText(/../“Error detecting Text $it”/../) }
  44. None
  45. Custom Tensorflow Models W orks offline

  46. TensorFlow

  47. None
  48. None
  49. Retrain existing model 0 mobilenet_v1

  50. What if I could tell what kind of chips I

    was eating?
  51. Gather Training Data FFMPEG Folders of Images Retrain with new

    images Optimize for mobile Embed in app Store in Firebase App uses model
  52. Gather training data

  53. Export to Images using ffmpeg ffmpeg -i flings.mp4 flings/flings_%04d.jpg

  54. Folders of images

  55. Retrain with new images python -m scripts.retrain \ --bottleneck_dir=tf_files/bottlenecks \

    --how_many_training_steps=500 \ --model_dir=tf_files/models/ \ --summaries_dir=tf_files/training_summaries/"${ARCHITECTURE}" \ --output_graph=tf_files/retrained_graph.pb \ --output_labels=tf_files/retrained_labels.txt \ --architecture="${ARCHITECTURE}" \ --image_dir=training_data/south_african_chips
  56. Optimize for mobile bazel-bin/tensorflow/contrib/lite/toco/toco \ --input_file=AgencyDay/retrained_graph.pb \ --output_file=AgencyDay/chips_optimized_graph.tflite \ --input_format=TENSORFLOW_GRAPHDEF

    \ --output_format=TFLITE \ --input_shape=1,${IMAGE_SIZE},${IMAGE_SIZE},3 \ --input_array=input \ --output_array=final_result \ --inference_type=FLOAT \ --input_data_type=FLOAT
  57. Insert into App https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/ https://github.com/googlecodelabs/tensorflow-for-poets-2

  58. Nik Nak 1 Or Not 2 bit.ly/mlkit-riggaroo

  59. What if our model changes?

  60. Ship an app update 3 And hope that people download

    it 4
  61. Host on Firebase Updates automatically downloaded

  62. val cloudSource = FirebaseCloudModelSource.Builder("my_cloud_model") .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() FirebaseModelManager.getInstance() .registerCloudModelSource(cloudSource)

    ……
  63. g.co/codelabs/mlkit-android-custom-model

  64. You don’t need to be a ML Expert to take

    advantage of ML in your apps!
  65. Thank you!

  66. Resources - https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets - https://codelabs.developers.google.com/ codelabs/tensorflow-for-poets-2-tflite/ - https://codelabs.developers.google.com/ codelabs/mlkit-android-custom-model/

    #0 - https://github.com/riggaroo/android- demo-mlkit