On-device machine learning in Flutter

Today we build a Flutter app that uses a neural network to apply some amazing style transformations on an image. The focus of the talk is mainly on understanding a few basic things about ML, ONNX (which just like Flutter is a cross-platform ML runtime) and to give you a jumpstart if you want to start integrating ML in your app. Tips and tricks included.

Transcript

1. FLUTTER ONNX RUNTIME Andrei Diaconu Microsoft
2. Andrei Diaconu 2010 – Android Android Iasi with Andrei Verdes 2018 – Flutter Wrote PostMuse using Flutter 2020 – Microsoft Developer advocate in the Flutter community This is a story, not a lecture.
3. Machine Learning What is it? What does it excel at? How do you implement it? What is inference and how do you do it in Flutter?
4. ML Frameworks • Training data • Training algorithms • Operators • Inference Model ONNX can be compared to a programming language specialized in mathematical functions.
5. ORT + Pieces Article • TensorFlow Lite is more restrictive. • ORT offers a C library. • Quantization • Continuous model refinement Article: ONNX Runtime: performant on-device inferencing - Microsoft Open Source Blog The app: https://pieces.app
6. LET’S FIRST GO TO THE PUB Tip: Easiest way to contribute to OSS is to solve your own issues and upstream the changes.
7. WHICH ONE DO YOU PICK? Tip: Whenever you decide which library to use, have a look at the code and the issues raised on GitHub. This gives you a better picture of overall quality and completeness. ?
8. Run the sample app A good example for audio processing. It is far too complicated. I am still learning. I need something simpler. Link: https://github.com/gtbluesky/onnxrunti me_flutter/tree/main/example
9. STEP 1 THE SIMPLEST MODEL Tip: When setting up ONNX Runtime, use a simple model. It eliminates issues that stem from processing, model complexity, supported operators, and so on
10. Link: onnxruntime-inference-examples/mobile/examples/basic_usage/model at main · microsoft/onnxruntime-inference-examples · GitHub
11. Link: onnxflutterplay/lib/main.dart at main · andreidiaconu/onnxflutterplay · GitHub Load model A and B (input tensors) Run model C (output tensor)
12. Link: onnxflutterplay/lib/main.dart at main · andreidiaconu/onnxflutterplay · GitHub
13. STEP 2 COMPLEX MODEL Tip: Install Netron so you can simply double- click to view on your PC. Always check the inputs and outputs this way to avoid confusion.
14. Link: examples/fast_neural_style at main · pytorch/examples · GitHub
15. Link: models/validated/vision/style_transfer/fast_neural_style at main · onnx/models · GitHub
16. STEP 2 COMPLEX MODEL Tensor shape?
17. STEP 1 THE SIMPLEST MODEL
18. CONVERT ONNX TO ORT Link: ORT model format | onnxruntime
19. What images to use Tip: Working with images means very large arrays, which are hard to follow. Whenever some input or output is hard to debug, ask yourself what image you can manufacture to help find the problem. Red 224x224 Red-Green 224x224
20. PRE-PROCESSING POST-PROCESSING P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Example image,1X10 all red Flutter raw image format: RBGA RGBA RGBA … R1 G1 B1 A1 R2 G2 B2 A2 … … … Tensor: RRRRR GGGGG BBBBB (no alpha) R1 R2 R3 R4 … … R10 G1 G2 G3 …
21. PRE-PROCESSING POST-PROCESSING P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Example image,1X10 all red Flutter raw image format: RBGA RGBA RGBA … 255 0 0 255 255 0 0 255 255 0 0 Tensor: RRRRR GGGGG BBBBB (no alpha) 255 255 255 255 … … 255 0 0 0 …
22. pre-processing
23. post-processing
24. RUN POST-PROCESSING ON PRE-PROCESSING RESULTS Tip: Working with images? Feed your pre-processing to your post-processing and display it on the screen. This makes many issues easy to spot. 255 255 255 255 … … 255 0 0 0 … …
25. RESULTS Tip: Take the smallest step you can think of. There is a lot that can go wrong when processing large tensors such as those for images. Creating bespoke images to use as input is also a skill you need to learn. Photo of an Eurasian blue tit by Francis Franklin
26. Resources These slides https://speakerdeck.com/andreidiaconu Code https://github.com/andreidiaconu/onnxflutterplay Article https://devblogs.microsoft.com/surface-duo/flutter-onnx- runtime/