FLUTTER ONNX RUNTIME Andrei Diaconu Microsoft 

Andrei Diaconu 2010 – Android Android Iasi with Andrei Verdes 2018 – Flutter Wrote PostMuse using Flutter 2020 – Microsoft Developer advocate in the Flutter community This is a story, not a lecture. 

Machine Learning What is it? What does it excel at? How do you implement it? What is inference and how do you do it in Flutter? 

ML Frameworks • Training data • Training algorithms • Operators • Inference Model ONNX can be compared to a programming language specialized in mathematical functions. 

ORT + Pieces Article • TensorFlow Lite is more restrictive. • ORT offers a C library. • Quantization • Continuous model refinement Article: ONNX Runtime: performant on-device inferencing - Microsoft Open Source Blog The app: https://pieces.app 

LET’S FIRST GO TO THE PUB Tip: Easiest way to contribute to OSS is to solve your own issues and upstream the changes. 

WHICH ONE DO YOU PICK? Tip: Whenever you decide which library to use, have a look at the code and the issues raised on GitHub. This gives you a better picture of overall quality and completeness. ? 

Run the sample app A good example for audio processing. It is far too complicated. I am still learning. I need something simpler. Link: https://github.com/gtbluesky/onnxrunti me_flutter/tree/main/example 

STEP 1 THE SIMPLEST MODEL Tip: When setting up ONNX Runtime, use a simple model. It eliminates issues that stem from processing, model complexity, supported operators, and so on 

Link: onnxruntime-inference-examples/mobile/examples/basic_usage/model at main · microsoft/onnxruntime-inference-examples · GitHub 

Link: onnxflutterplay/lib/main.dart at main · andreidiaconu/onnxflutterplay · GitHub Load model A and B (input tensors) Run model C (output tensor) 

Link: onnxflutterplay/lib/main.dart at main · andreidiaconu/onnxflutterplay · GitHub 

STEP 2 COMPLEX MODEL Tip: Install Netron so you can simply double- click to view on your PC. Always check the inputs and outputs this way to avoid confusion. 

Link: examples/fast_neural_style at main · pytorch/examples · GitHub 

Link: models/validated/vision/style_transfer/fast_neural_style at main · onnx/models · GitHub 

STEP 2 COMPLEX MODEL Tensor shape? 

STEP 1 THE SIMPLEST MODEL 

CONVERT ONNX TO ORT Link: ORT model format | onnxruntime 

What images to use Tip: Working with images means very large arrays, which are hard to follow. Whenever some input or output is hard to debug, ask yourself what image you can manufacture to help find the problem. Red 224x224 Red-Green 224x224 

PRE-PROCESSING POST-PROCESSING P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Example image,1X10 all red Flutter raw image format: RBGA RGBA RGBA … R1 G1 B1 A1 R2 G2 B2 A2 … … … Tensor: RRRRR GGGGG BBBBB (no alpha) R1 R2 R3 R4 … … R10 G1 G2 G3 … 

PRE-PROCESSING POST-PROCESSING P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 Example image,1X10 all red Flutter raw image format: RBGA RGBA RGBA … 255 0 0 255 255 0 0 255 255 0 0 Tensor: RRRRR GGGGG BBBBB (no alpha) 255 255 255 255 … … 255 0 0 0 … 

pre-processing 

post-processing 

RUN POST-PROCESSING ON PRE-PROCESSING RESULTS Tip: Working with images? Feed your pre-processing to your post-processing and display it on the screen. This makes many issues easy to spot. 255 255 255 255 … … 255 0 0 0 … … 

RESULTS Tip: Take the smallest step you can think of. There is a lot that can go wrong when processing large tensors such as those for images. Creating bespoke images to use as input is also a skill you need to learn. Photo of an Eurasian blue tit by Francis Franklin 

Resources These slides https://speakerdeck.com/andreidiaconu Code https://github.com/andreidiaconu/onnxflutterplay Article https://devblogs.microsoft.com/surface-duo/flutter-onnx- runtime/