Applied Firebase ML Kit

Learn about the exciting new Firebase MLKit SDK, discover how top apps are leveraging it to solve real-world problems, and follow along as Tyler walks us through how he used ML Kit to create a real version of the "Hotdog / Not-Hotdog" app from the popular Silicon Valley TV Series on HBO.

Tech talk given at https://dcnyc-extended-2018.splashthat.com/

Transcript

1. Applied Firebase ML Kit Tyler Eastman - Walmart Labs @tyler5673
3. Applied Firebase ML Kit Tyler Eastman - Walmart Labs @tyler5673
4. Machine learning is HARD • Heavily math based • It's relatively New!
5. Super brief timeline • 1950s - Research is conducted with simple algorithms • 1970s - 'AI Winter' about machine learning effectiveness • 1990s - Shift from a knowledge-driven to a data-driven approach • 2006 - Netflix Prize • 2010 - Deep learning becomes feasible • 2012 - Cats in youtube videos
6. From TensorFlow Lite for mobile developers (Google I/O '18)
7. Where is it used? • Google (duh!) • Medical Research • Self-driving cars • etc ...
8. Ok, but how does a neural network... work?
9. From Machine learning models + IoT data = a smarter world (Google I/O '18)
10. Tensorflow • Open source • Works on CPUs and GPUs • Huge complex models • Best for cloud applications • 'tensor' is a multi-dimensional data array
12. Tensorflow Lite • Lightweight solution for mobile devices • Android Only initially, cross platform just announced • Hardware accelerated • Supports a subset of TF models • Best solution for running on your device!
13. • Pros • Latency • Works when not connected • Pricing • Privacy (data stays on device) • (More) battery efficient • Can take advantage of the sensor data right on the device
14. • Cons • Tight memory constraints • Less processing power • Less complex models • Sorry
15. From TensorFlow Lite for mobile developers (Google I/O '18)
16. From TensorFlow Lite for mobile developers (Google I/O '18)
19. ML Kit!
20. From Introducing ML Kit (Google I/O '18')
21. Some cool things • On (and/or) Off Device ML! • Clean API to TF Lite • No latency woes • Privacy
22. Now powered by Firebase! • Model hosting • A/B comparisons • Performance monitoring • Analytics
23. ML Kit Features From https://firebase.google.com/docs/ml-kit/
24. Work smart NOT hard — Me
25. You miss %100 of the shots you don't take — Wayne Gretsky
26. You miss %100 of the shots you don't take — Wayne Gretsky Michael Scott
27. How do I integrate it into my app?
28. Open Firebase Assistant
29. Open Firebase Assistant
30. Connect to Firebase
33. Hotdog, Not-Hotdog
35. Implementing Vision Image Label API
36. Add your dependencies dependencies { // ... implementation 'com.google.firebase:firebase-ml-vision:17.0.0' implementation 'com.google.firebase:firebase-ml-vision-image-label-model:15.0.0' }
37. Optional (but recommended) Add this to your manifest to download model on install ... otherwise it will be downloaded on first use <application ...> ... <meta-data android:name="com.google.firebase.ml.vision.DEPENDENCIES" android:value="label" /> <!-- To use multiple models: android:value="label,model2,model3" --> </application>
38. To the code!
39. 1) Configure the image labeler // On Device FirebaseVisionLabelDetectorOptions options = new FirebaseVisionLabelDetectorOptions.Builder() .setConfidenceThreshold(0.8f) .build(); // Cloud FirebaseVisionCloudDetectorOptions options = new FirebaseVisionCloudDetectorOptions.Builder() .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL) .setMaxResults(15) .build();
40. 2) Create FirebaseVisionImage object // This FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap); // Or this FirebaseVisionImage image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation); // Or (yuck) even this FirebaseVisionImage image = FirebaseVisionImage.fromByteBuffer(buffer, metadata);
41. 3) Get an instance of FirebaseVisionLabelDetector FirebaseVisionLabelDetector detector = FirebaseVision.getInstance() .getVisionLabelDetector(); // Or, to set the minimum confidence required: FirebaseVisionLabelDetector detector = FirebaseVision.getInstance() .getVisionLabelDetector(options);
42. 4) Finally, pass the image to the detector Task<List<FirebaseVisionLabel>> result = detector.detectInImage(image) .addOnSuccessListener( new OnSuccessListener<List<FirebaseVisionLabel>>() { @Override public void onSuccess(List<FirebaseVisionLabel> labels) { // Task completed successfully // ... } }) .addOnFailureListener( new OnFailureListener() { @Override public void onFailure(@NonNull Exception e) { // Task failed with an exception // ... } });
43. How do we inspect the labels? for (FirebaseVisionLabel label: labels) { String text = label.getLabel(); String entityId = label.getEntityId(); float confidence = label.getConfidence(); }
44. But what about a Hotdog? val isHotdog = labels.any { it.getLabel() == "hotdog" }
45. Demo!
49. Q&A
50. Thank you! @tyler5673