Flutter + ML Kit: Building smart cross platform apps

This was a talk given at #PhillyETE 2019.

Here's the YouTube video: https://bit.ly/phillyete19_flutter_mlkit_video

Here's the abstract:

Getting started with machine learning on the mobile platform can be daunting. From learning how to implement low-level models and use frameworks, to optimizing and adapting those models for mobile, it can be overwhelming. With ML Kit, regardless of one’s machine learning expertise, one can build compelling machine learning functionality into their mobile apps. Using ML Kit and Flutter, which is Google’s mobile SDK for building native Android and iOS apps from a single codebase, one can build apps with machine learning capabilities like face detection and image labeling, for both Android and iOS platforms in record time!

This talk will be guided by the following objectives:
- An introduction to Flutter and to ML Kit
- Determining how to use Flutter to leverage ML Kit’s functionalities
- Exploring ML Kit’s on-device features that are currently available, which are text recognition, face detection, barcode scanning and image labeling.
- Equipping attendees with the knowledge they need to build a smart Flutter application with ML Kit.

#flutter #mlkit #machinelearning

Transcript

1. WIFI
 ETE2019 Presented By HASHTAG #PhillyETE SESSION Q&A Visit sli.do
 #PHILLYETE
2. Flutter + ML Kit Building smart cross-platform applications Ifeoma Okereke Sr Software Engineer @ WW (formerly known as Weight Watchers) @ifeomaro
3. Objectives • Flutter ◦ Intro to Flutter ◦ Flutter widgets ◦ Explore Flutter demo app • ML Kit ◦ Intro to ML Kit ◦ Explore ML Kit APIs
4. Flutter
5. What is Flutter? Google’s mobile SDK, used to create beautiful, native interfaces on both iOS and Android, from a single codebase, in record time
6. What’s great about Flutter • Fast cross-platform development
7. What’s great about Flutter • Fast cross-platform development • Performant native experience
8. What’s great about Flutter • Fast cross-platform development • Performant native experience • Lots of built-in widgets, not limited to OEM widgets
9. What’s great about Flutter • Fast cross-platform development • Performant native experience • Lots of built-in widgets, not limited to OEM widgets • Platform specific code where needed
10. What’s great about Flutter • Fast cross-platform development • Performant native experience • Lots of built-in widgets, not limited to OEM widgets • Platform specific code where needed • Android Jelly Bean, API 16 or newer, and iOS 8 or newer.
11. What’s great about Flutter • Fast cross-platform development • Performant native experience • Lots of built-in widgets, not limited to OEM widgets • Platform specific code where needed • Android Jelly Bean, API 16 or newer, and iOS 8 or newer. • Hot reload
13. What’s great about Flutter • Using Dart
14. What’s great about Flutter • Using Dart ◦ Ahead of time (AOT) compilation
15. What’s great about Flutter • Using Dart ◦ Ahead of time (AOT) compilation ◦ Just in time (JIT) compilation
16. http://bit.ly/magic_of_flutter
18. How do I get started?
19. How do I get started? • IDE ◦ Android Studio ◦ IntelliJ IDEA ◦ Visual Studio Code ◦ Also, command line
20. How do I get started? • IDE ◦ Android Studio ◦ IntelliJ IDEA ◦ Visual Studio Code ◦ Also, command line • Download Dart and Flutter plugins
21. How do I get started? • IDE ◦ Android Studio ◦ IntelliJ IDEA ◦ Visual Studio Code ◦ Also, command line • Download Dart and Flutter plugins • Download the Flutter SDK ◦ flutter.io/docs/get-started/install
22. How do I get started? • IDE ◦ Android Studio ◦ IntelliJ IDEA ◦ Visual Studio Code ◦ Also, command line • Download Dart and Flutter plugins • Download the Flutter SDK ◦ flutter.io/docs/get-started/install • Create a Flutter project
37. Widgets
38. Widgets • Basic building blocks of a Flutter app’s UI
39. Widgets • Basic building blocks of a Flutter app’s UI • A widget can be:
40. Widgets • Basic building blocks of a Flutter app’s UI • A widget can be: ◦ a structural element (button)
41. Widgets • Basic building blocks of a Flutter app’s UI • A widget can be: ◦ a structural element (button) ◦ an aspect of layout (padding)
42. Widgets • Basic building blocks of a Flutter app’s UI • A widget can be: ◦ a structural element (button) ◦ an aspect of layout (padding) ◦ a navigation component
43. Almost everything in Flutter is a widget!
44. Stateless & Stateful widgets
45. Stateless & Stateful widgets If a widget can change, it's stateful, otherwise it's stateless
46. Stateless widgets
47. Stateful widgets • Dynamic • Change their appearance in response to user interactions or changes in data
49. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
50. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
51. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
52. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
53. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
54. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
55. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
56. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
57. import 'package:flutter/material.dart'; void main() => runApp(MyApp()); class MyApp extends StatelessWidget { @override Widget build(BuildContext context) { return MaterialApp( title: 'Flutter Demo', theme: ThemeData( primarySwatch: Colors.pink, ), home: MyHomePage(title: 'Philly ETE Demo'), ); } } class MyHomePage extends StatefulWidget { MyHomePage({Key key, this.title}) : super(key: key); final String title; @override _MyHomePageState createState() => _MyHomePageState(); }
58. class _MyHomePageState extends State<MyHomePage> { int _count = 0; void _incrementCount() { setState(() { ++_count; }); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text(widget.title), ), body: Center( child: Text( '$_count', style: TextStyle(fontWeight: FontWeight.bold, fontSize: 100), ), ), floatingActionButton: FloatingActionButton( onPressed: _incrementCount, tooltip: 'Increment', child: Icon(Icons.add), ),
59. class _MyHomePageState extends State<MyHomePage> { int _count = 0; void _incrementCount() { setState(() { ++_count; }); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text(widget.title), ), body: Center( child: Text( '$_count', style: TextStyle(fontWeight: FontWeight.bold, fontSize: 100), ), ), floatingActionButton: FloatingActionButton( onPressed: _incrementCount, tooltip: 'Increment', child: Icon(Icons.add), ),
60. class _MyHomePageState extends State<MyHomePage> { int _count = 0; void _incrementCount() { setState(() { ++_count; }); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text(widget.title), ), body: Center( child: Text( '$_count', style: TextStyle(fontWeight: FontWeight.bold, fontSize: 100), ), ), floatingActionButton: FloatingActionButton( onPressed: _incrementCount, tooltip: 'Increment', child: Icon(Icons.add), ),
61. class _MyHomePageState extends State<MyHomePage> { int _count = 0; void _incrementCount() { setState(() { ++_count; }); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text(widget.title), ), body: Center( child: Text( '$_count', style: TextStyle(fontWeight: FontWeight.bold, fontSize: 100), ), ), floatingActionButton: FloatingActionButton( onPressed: _incrementCount, tooltip: 'Increment', child: Icon(Icons.add), ),
62. class _MyHomePageState extends State<MyHomePage> { int _count = 0; void _incrementCount() { setState(() { ++_count; }); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text(widget.title), ), body: Center( child: Text( '$_count', style: TextStyle(fontWeight: FontWeight.bold, fontSize: 100), ), ), floatingActionButton: FloatingActionButton( onPressed: _incrementCount, tooltip: 'Increment', child: Icon(Icons.add), ),
63. class _MyHomePageState extends State<MyHomePage> { int _count = 0; void _incrementCount() { setState(() { ++_count; }); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text(widget.title), ), body: Center( child: Text( '$_count', style: TextStyle(fontWeight: FontWeight.bold, fontSize: 100), ), ), floatingActionButton: FloatingActionButton( onPressed: _incrementCount, tooltip: 'Increment', child: Icon(Icons.add), ),
64. class _MyHomePageState extends State<MyHomePage> { int _count = 0; void _incrementCount() { setState(() { ++_count; }); } @override Widget build(BuildContext context) { return Scaffold( appBar: AppBar( title: Text(widget.title), ), body: Center( child: Text( '$_count', style: TextStyle(fontWeight: FontWeight.bold, fontSize: 100), ), ), floatingActionButton: FloatingActionButton( onPressed: _incrementCount, tooltip: 'Increment', child: Icon(Icons.add), ),
66. How does Flutter compare with other cross-platform solutions?
67. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514 Native application Services
68. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514 Webview application
69. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514 Reactive application
70. https://hackernoon.com/whats-revolutionary-about-flutter-946915b09514 Flutter application
71. Cons of Flutter • Actively being developed
73. Cons of Flutter • Minimum app size ◦ Android: 4.08 MB ◦ iOS: 10.8 MB
74. Cons of Flutter • Minimum app size ◦ Android: 4.08 MB ◦ iOS: 10.8 MB
75. In summary … • Intro to Flutter • What’s great about Flutter • Flutter Demo app • Widgets: stateless vs stateful • How Flutter compares to other cross-platform solutions • Cons of Flutter
77. ML Kit
79. • Machine Learning Kit
80. • Machine Learning Kit • Provided by Google via Firebase
81. • Machine Learning Kit • Provided by Google via Firebase • Provides prebuilt or custom machine learning models for a variety of use cases
82. • Machine Learning Kit • Provided by Google via Firebase • Provides prebuilt or custom machine learning models for a variety of use cases • Cross-platform: available on Android, iOS and Flutter
83. • Machine Learning Kit • Provided by Google via Firebase • Provides prebuilt or custom machine learning models for a variety of use cases • Cross-platform: available on Android, iOS and Flutter • Runs on-device or on the cloud
84. • Machine Learning Kit • Provided by Google via Firebase • Provides prebuilt or custom machine learning models for a variety of use cases • Cross-platform: available on Android, iOS and Flutter • Runs on-device or on the cloud • Runs free on-device; partly free on the cloud
85. ML Kit APIs Barcode scanning Landmark recognition Image labeling Text recognition Custom models Face detection
86. ML Kit API Image Data Image/Video
87. Image labeling
88. Image labeling • Recognize the entities in an image without providing any additional context
89. Image labeling • Recognize the entities in an image without providing any additional context • Can be used for automatic metadata generation or content moderation
90. Image labeling • Recognize the entities in an image without providing any additional context • Can be used for automatic metadata generation or content moderation • Free, on-device, and on the cloud (for the first 1000 uses)
91. Image Labeling API • Flowerpot, 0.84 • Desk, 0.64 • Mobile phone, 0.64 • Television, 0.56 • Tableware, 0.54 • Plant, 0.52 Image ImageLabels, Confidence ratio https://unsplash.com/photos/O5mmDoBqTF8
92. final FirebaseVisionImage visionImage = FirebaseVisionImage.fromFile(imageFile); final FirebaseVisionDetector detector = FirebaseVision.instance.labelDetector(); final Future<List<Label>> results = await detector.detectInImage(visionImage) ?? <Label>[];
93. Label • Confidence ratio • Label • Entity ID Image labeling
95. Face detection
96. Face detection • Locate human faces in an image/video
97. Face detection • Locate human faces in an image/video • Identify key facial features
98. Face detection • Locate human faces in an image/video • Identify key facial features • Get the contours of detected faces
99. Face detection • Locate human faces in an image/video • Identify key facial features • Get the contours of detected faces • Recognize facial expressions
100. Face detection • Locate human faces in an image/video • Identify key facial features • Get the contours of detected faces • Recognize facial expressions • Track faces across video frames
101. final FirebaseVisionDetector detector = FirebaseVision.instance.faceDetector( const FaceDetectorOptions( enableClassification: true, enableLandmarks: true, enableTracking: true, mode: FaceDetectorMode.accurate, ), );
102. final Future<List<Face>> results = await detector.detectInImage(visionImage) ?? <Face>[];
103. Face • Bounding box • Landmarks • Left Eye Open Probability • Right Eye Open Probability • Smiling Probability Landmarks • Landmark type: left_eye, right_eye, left_ear, right_ear • Position Face detection
105. Text recognition
106. Text recognition • Recognize texts in images
107. Text recognition • Recognize texts in images • On-device and on the cloud
108. Text recognition • Recognize texts in images • On-device and on the cloud • On-device only supports Latin-based languages
109. final FirebaseVisionImage visionImage = FirebaseVisionImage.fromFile(imageFile); final FirebaseVisionDetector detector = FirebaseVision.instance.textRecognizer(); final Future<VisionText> results = await detector.detectInImage(visionImage) ?? <VisionText>[];
110. VisionText • Text • Text Block Text recognition
111. VisionText • Text • Text Block Text recognition Text Block • Text • Text Line • Language • Bounding box • Confidence
112. VisionText • Text • Text Block Text recognition Text Block • Text • Text Line • Language • Bounding box • Confidence Text Line • Text • Text Element • Language • Bounding box • Confidence
113. VisionText • Text • Text Block Text recognition Text Block • Text • Text Line • Language • Bounding box • Confidence Text Line • Text • Text Element • Language • Bounding box • Confidence Text Element • Text • Language • Bounding box • Confidence
115. Barcode scanning
116. Barcode scanning • Read data encoded using most standard barcode formats.
117. Barcode scanning • Read data encoded using most standard barcode formats. • Works with any image orientation
118. final FirebaseVisionImage visionImage = FirebaseVisionImage.fromFile(imageFile); final FirebaseVisionDetector detector = FirebaseVision.instance.barcodeDetector(); final Future<List<Barcode>> results = await detector.detectInImage(visionImage) ?? <Barcode>[];
119. Barcode • Value type • Raw value • Display value • Bounding box Barcode scanning
121. Landscape detection
122. Landscape detection • Recognize well-known landmarks in an image
123. Landscape detection • Recognize well-known landmarks in an image • API provides: ◦ name of the landmarks recognized ◦ its geographic coordinates ◦ the region of the image the landmark was found
124. Custom models
125. Custom models • Create models using TensorFlow Lite
126. Custom models • Create models using TensorFlow Lite • Host your models using Firebase or package them in your app
127. Custom models • Create models using TensorFlow Lite • Host your models using Firebase or package them in your app • Use the ML Kit SDK to perform inference of your custom model
128. In summary ... • ML Kit APIs • Image labeling • Face detection • Text recognition • Barcode scanning • Landmark detection • Custom models
129. Resources • Documentation: flutter.dev/docs • Flutter for Android Devs: flutter.dev/docs/get-started/ flutter-for/android-devs • Flutter for iOS Devs: flutter.dev/docs/get-started/ flutter-for/ios-devs • Udacity Flutter course: classroom.udacity.com/ courses/ud905 • Codelabs: flutter.io/docs/codelabs • ML Kit library: https://pub.dartlang.org/packages/ firebase_ml_vision #89
130. We’re hiring! ww.com/careers
131. Thanks for listening! @IfeomaRO