Flutter + MLKit: Building smart cross-platform applications

These are the slides for my presentation at Droidcon San Francisco 2018.

Google slides presentation here: https://bit.ly/dcsf18_flutter_mlkit

Youtube video of the talk: https://bit.ly/dcsf18_flutter_mlkit_video

Talk proposal:
Getting started with machine learning on the mobile platform can be daunting. From learning how to implement low-level models and use frameworks, to optimizing and adapting those models for mobile, it can be overwhelming. With ML Kit, regardless of one’s machine learning expertise, one can build compelling machine learning functionality into their mobile apps. Using ML Kit and Flutter, which is Google’s mobile SDK for building native Android and iOS apps from a single codebase, one can build apps with machine learning capabilities like face detection and image labeling, for both Android and iOS platforms in record time!

This talk will be guided by the following objectives:
- An introduction to Flutter and to ML Kit
- Exploration of some of the key functionalities that ML Kit offers such as text recognition and landmark recognition.
- Determining how to use Flutter to leverage ML Kit’s functionalities
- Exploring ML Kit’s on-device features that are currently available, which are text recognition, face detection, barcode scanning and image labeling.
- Equipping attendees with the knowledge they need to build a smart Flutter application with ML Kit.

Transcript

1. Flutter + MLKit Building smart cross-platform applications Ifeoma Okereke Android Engineer @ WW (formerly known as Weight Watchers) @IfeomaRO
2. Objectives • Flutter ◦ Intro ◦ Writing a Flutter app ◦ Why use Flutter • ML Kit ◦ Intro ◦ ML Kit APIs
3. Flutter
4. What is Flutter? • Google’s mobile SDK • Create native interfaces on both iOS and Android • Single codebase • Build cross-platform apps in record time • Using Dart
5. What’s great about Flutter • Fast cross-platform development • Performant native experience • Lots of built-in widgets, not limited to OEM widgets • Platform specific code where needed • Android Jelly Bean, API 16 or newer, and iOS 8 or newer.
6. What’s great about Flutter • Hot reload • Using Dart
7. http://bit.ly/magic_of_flutter
8. How do I get started? • IDE ◦ Android Studio ◦ IntelliJ IDEA ◦ Visual Studio Code ◦ Command line • Download Dart and Flutter plugins • Download the Flutter SDK ◦ flutter.io/docs/get-started/install
9. How do I get started? • Create a Flutter project
19. Widgets • Basic building blocks of a Flutter app’s UI • Immutable: all fields must be final • It can be ◦ a structural element (button) ◦ a stylistic element (font) ◦ an aspect of layout (padding) ◦ a navigation component
20. Stateless vs Stateful Stateless • all values are final • does not require a mutable state Stateful • requires a mutable state • maintain state that may change during the app’s lifecycle
27. Hot reload • Similar to instant run but faster (less than a second) and better
29. How does Flutter compare with other cross-platform solutions?
30. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514
31. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514
32. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514
33. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514
34. How does Flutter compare? • More performant since there is minimal bridge cost
36. ML Kit
37. • Machine Learning Kit • Provides prebuilt or custom machine learning models for a variety of use cases • Provided by Google via Firebase • Cross-platform: available on Android, iOS and Flutter • Runs on-device or on the cloud
38. ML Kit APIs • Image labeling • Text recognition • Face detection • Barcode scanning • Landmark recognition • Custom models
39. ML Kit API Image Data Image
40. Image labeling Recognize the entities in an image without providing any additional context
41. Image Labeling API • Flowerpot, 0.84 • Desk, 0.64 • Mobile phone, 0.64 • Television, 0.56 • Tableware, 0.54 • Plant, 0.52 Image Labels, Confidence ratio https://unsplash.com/photos/O5mmDoBqTF8
42. final FirebaseVisionImage visionImage = FirebaseVisionImage.fromFile(imageFile); final FirebaseVisionDetector detector = FirebaseVision.instance.labelDetector(); final Future<List<Label>> results = await detector.detectInImage(visionImage) ?? <Label>[];
43. Label • Confidence • Label • Entity ID Image labeling
44. Face detection Locate human faces in an image/video
45. final FirebaseVisionDetector detector = FirebaseVision.instance.faceDetector( const FaceDetectorOptions( enableClassification: true, enableLandmarks: true, enableTracking: true, mode: FaceDetectorMode.accurate, ), );
46. final FirebaseVisionDetector detector = FirebaseVision.instance.faceDetector( const FaceDetectorOptions( enableClassification: true, enableLandmarks: true, enableTracking: true, mode: FaceDetectorMode.accurate, ), );
47. final FirebaseVisionDetector detector = FirebaseVision.instance.faceDetector( const FaceDetectorOptions( enableClassification: true, enableLandmarks: true, enableTracking: true, mode: FaceDetectorMode.accurate, ), );
48. final FirebaseVisionDetector detector = FirebaseVision.instance.faceDetector( const FaceDetectorOptions( enableClassification: true, enableLandmarks: true, enableTracking: true, mode: FaceDetectorMode.accurate, ), );
49. final FirebaseVisionDetector detector = FirebaseVision.instance.faceDetector( const FaceDetectorOptions( enableClassification: true, enableLandmarks: true, enableTracking: true, mode: FaceDetectorMode.accurate, ), );
50. final Future<List<Face>> results = await detector.detectInImage(visionImage) ?? <Face>[];
51. Face • Bounding box • Landmarks • Left Eye Open Probability • Right Eye Open Probability • Smiling Probability Landmarks • Landmark type: left_eye, right_eye, left_ear, right_ear • Position Face detection
52. Text recognition Recognize texts in images
53. final FirebaseVisionImage visionImage = FirebaseVisionImage.fromFile(imageFile); final FirebaseVisionDetector detector = FirebaseVision.instance.textRecognizer(); final Future<VisionText> results = await detector.detectInImage(visionImage) ?? <VisionText>[];
54. VisionText • Text • Text Block Text recognition Text Block • Text • Text Line • Language • Bounding box • Confidence Text Line • Text • Text Element • Language • Bounding box • Confidence Text Element • Text • Language • Bounding box • Confidence
55. Barcode scanning Read data encoded using most standard barcode formats.
56. final FirebaseVisionImage visionImage = FirebaseVisionImage.fromFile(imageFile); final FirebaseVisionDetector detector = FirebaseVision.instance.barcodeDetector(); final Future<List<Barcode>> results = await detector.detectInImage(visionImage) ?? <Barcode>[];
57. Barcode • Value type • Raw value • Display value • Bounding box Barcode scanning
58. Landscape detection • Recognize well-known landmarks in an image • API provides: ◦ name of the landmarks recognized ◦ its geographic coordinates ◦ the region of the image the landmark was found
59. Custom models • Host your models using Firebase or package them in your app • Host models using Firebase to reduce your app’s binary size and to ensure that your app always uses the latest version of your model
60. Custom models • Use the ML Kit SDK to perform inference using the best available version of your custom model • Configure automatic update conditions for your model
61. In summary ... • Building cross-platform apps with Flutter • ML Kit APIs
62. Resources • Documentation: flutter.io/docs • Flutter for Android Devs: flutter.io/docs/get-started/flutter-for/android-devs • Udacity Flutter course: classroom.udacity.com/courses/ud905 • Codelabs: flutter.io/docs/codelabs • ML Kit library: https://pub.dartlang.org/packages/firebase_ml_vision
63. Thanks for listening! @IfeomaRO
64. We’re hiring! ww.com/careers