Firebase MLKit : Machine Learning Made Easy

Transcript

1. Firebase MLKit Machine Learning Made Easy 1 Harshit Dwivedi, Android Instructor & Developer @CodingBlocks
2. HELLO! I am An Android Developer and Open Source enthusiast who loves to make better and beautiful apps. 2
3. Machine Learning as a Mobile Developer 3
5. 5 Me right now
6. What do ? :| 6
7. Enter Firebase MLKit 7
8. Firebase What ?! 8 • Announced by Google at I/O 2018 • Contains prebuilt APIs which provide ML functionalities for Android and iOS apps • Support for running custom models
9. What all do I need to start? 9 • An app/idea that needs machine learning features • Basic knowledge of Python and Tensorflow (for custom models) • Basics of App Development, Duh!?
10. Available APIs 10 • Text Recognition • Face Detection • Barcode Scanning • Image Labeling • Landmark Recognition • Hosting custom Tensorflow models
11. Type of APIs 11 • On Device ◦ Free ◦ Runs without Internet ◦ Low Accuracy • Cloud ◦ Paid (free for first 1000 calls per month) ◦ Needs internet connectivity ◦ High Accuracy
12. MLKit in Action 12
13. 1. Text Recognition 13 • Extract text from images • Cloud and On Device APIs available
14. Steps Involved 14 1. Create a FirebaseVisionImage from the Bitmap
16. Steps Involved 16 1. Create a FirebaseVisionImage from the Bitmap 2. Get access to the cloud or on device text detector
18. Steps Involved 18 1. Create a FirebaseVisionImage from the Bitmap 2. Get access to the cloud or on device text detector 3. Run the detector on the image
20. Steps Involved 20 1. Create a FirebaseVisionImage from the Bitmap 2. Get access to the cloud or on device text detector 3. Run the detector on the image 4. Extract text from the response
22. 2. Barcode Scanning 22 • Scan and process barcodes • On Device only API available
23. Steps Involved 23 1. Create a FirebaseVisionImage from the Bitmap
25. Steps Involved 25 1. Create a FirebaseVisionImage from the Bitmap 2. Determine the types of barcode you want to scan (optional)
27. Steps Involved 27 1. Create a FirebaseVisionImage from the Bitmap 2. Determine the types of barcode you want to scan 3. Get access to the on device barcode detector
29. Steps Involved 29 1. Create a FirebaseVisionImage from the Bitmap 2. Determine the types of barcode you want to scan 3. Get access to the on device barcode detector 4. Run the detector on the image
31. Steps Involved 31 1. Creating a FirebaseVisionImage from the Bitmap 2. Determine the types of barcode you want to scan 3. Getting access to the on device barcode detector 4. Running the detector on the image 5. Extract data from the response
33. 3. Image Labeling 33 • Identify objects, locations, activities, animal species, products, and more • Cloud and On Device APIs available
34. Steps Involved 34 1. Create a FirebaseVisionImage from the Bitmap 2. Specify the number of labels to return (optional) 3. Get access to the on device or cloud image label detector 4. Run the detector on the image 5. Extract data from the response
36. 4. Landmark Detection 36 • Identify popular landmarks in an image • Cloud only API available
37. Steps Involved 37 1. Create a FirebaseVisionImage from the Bitmap 2. Get access to the cloud landmark detector 3. Run the detector on the image 4. Extract data from the response
39. 5. Face Detection 39 • Detect faces and facial landmarks • On Device only API available
40. Steps Involved 40 1. Create a FirebaseVisionImage from the Bitmap 2. Specify the facial features you want to detect (optional) 3. Get access to the face detector 4. Run the detector on the image 5. Extract data from the response
42. 6. Custom Model / Case Study 42 • Host your TensorFlow Lite models using Firebase • ML Kit SDK automatically uses the best-available version of your custom model
43. Steps involved in training and running a Custom Model 43
44. 1. Creating a good Dataset 44 • Ensure that there are at least 150 images for every type of classification that you plan on doing
46. 1. Creating a good Dataset 46 • Ensure that there are at least 150 images for every type of classification that you plan on doing • Account for real life use cases by adding noise to your dataset
48. 1. Creating a good Dataset 48 • Ensure that there are at least 150 images for every type of classification that you plan on doing • Account for real life use cases by adding noise to your dataset • Augment your dataset by scaling, rotating and cropping images
50. 2. Training the Model 50 • Use Transfer Learning in Tensorflow to retrain a model
52. 2. Training the Model 52 • Use Transfer Learning in Tensorflow to retrain a model • Convert the trained model to a Tensorflow Lite Build
54. 3. Uploading model to Firebase 54 • Navigate to MLKit in Firebase Console • Select Custom Models and upload the model created in the last step
58. 4. Setup the model with Firebase 58 • Load the model (cloud or local)
60. 4. Setup the model with Firebase 60 • Load the model • Register the model with Firebase
62. 4. Setup the model with Firebase 62 • Load the model • Register the model with Firebase • Create the firebase interpreter for the model you loaded above
64. 5. Run the model over an image 64 • Convert the bitmap to a ByteBuffer
66. 5. Run the model over an image 66 • Convert the bitmap to a ByteBuffer • Specify the input and output options for the model
69. 5. Run the model over an image 69 • Convert the bitmap to a ByteBuffer • Specify the input and output options for the model • Run the interpreter over the image
71. 5. Run the model over an image 71 • Convert the bitmap to a ByteBuffer • Specify the input and output options for the model • Run the interpreter over the image • Extract data from the result
73. 6. Cry in Despair 73 • Things don’t always work as you want them to!
74. Tips for creating a good model 74 • Use lots of images (I used 10,000 images for the Pokémon model) • Augment your dataset by scaling, rotating and cropping images • Account for real life use cases by adding noise to your dataset
75. Takeaways 75 • Machine Learning is hard, but can be fun if done right!
76. Takeaways 76 • Machine Learning is hard, but can be fun if done right! • Tools and API available make the onboarding process super easy.
77. Takeaways 77 • Machine Learning is hard, but can be fun if done right! • Tools and API available make the onboarding process super easy. • For a custom model, having a good dataset should be your primary focus!
78. What next? 78 Try the Tensorflow for Poets Codelab • Gives you a sneak preview of what all is possible using Tensorflow. • Requires no prior knowledge of Python or Machine Learning https://codelabs.developers.google.com/codelabs/tensorflow-for-poets/
79. What next? 79 Go through Google’s Machine Learning Crash Course • Built for Developers with prior Programing experience • Uses Tensorflow to cover the basics of Machine Learning alongwith advance topics https://developers.google.com/machine-learning/crash-course/
80. What next? 80 Go ahead and experiment with the Firebase MLKit Codelabs • Short interactive coding tutorials that help you get up to speed on the MLKit APIs https://goo.gl/uPBNpB https://goo.gl/sFueGj https://goo.gl/XwTkoa https://goo.gl/QPstLM
81. What next? 81 Try the Fritz Mobile SDK for implementing complex Machine Learning models with relative ease. • Provides built in APIs similar to Firebase MLKit with some added APIs like Image and Video Styling. • Also allows you to host and run Custom Tensorflow Lite and Apple’s CoreML models. https://fritz.ai
82. Like what you saw? 82 Try out the MLKit app : https://github.com/the-dagger/MLKitAndroid/ Try out the MLKit Codelabs : https://goo.gl/dFsmRp
83. Like what you saw? 83 Try out PokeDex on Google Play store : https://goo.gl/xukauP Source code on Github : https://github.com/the-dagger/MLKitAndroid Dataset on Kaggle : https://kaggle.com/thedagger/pokemon-generation-one
84. THANKS! Questions? 84 You can find me on twitter at : https://twitter.com/daggerdwivedi My Medium blogs can be read at : https://medium.com/@daggerdwivedi My Github : https://github.com/the-dagger