Not Har Gow: Using Custom Models with ML Kit

Slide 1

Slide 1 text

Not Har Gow: Using Custom Models with ML Kit Eric Fung • @gnufmuffin Ohio DevFest • #OhioDevFest18 • 2018-10-13

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Demo

Slide 5

Slide 5 text

History Image credit: burst.shopify.com

Slide 6

Slide 6 text

TensorFlow ‣ Open-source framework for ML ‣ First release in Nov 2015 ‣ Included sample for Android Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 7

Slide 7 text

TensorFlow (Nov 2015) static { System.loadLibrary("tensorflow_demo"); } public native int initializeTensorflow(…) public native String classifyImageBmp(Bitmap bitmap) Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 8

Slide 8 text

TensorFlow (Jan 2017) // Load model from disk TensorFlowInferenceInterface tf = new TensorFlowInferenceInterface(assetManager, "my_model.pb"); // Copy the input data into TensorFlow tf.feed(inputName, floatValues, 1, inputSize, inputSize, 3); // Run the inference call. tf.run(outputNames, logStats); // Copy the output Tensor back into the output array. tf.fetch(outputName, outputs); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 9

Slide 9 text

TensorFlow for Mobile (May 2017) dependencies { compile 'org.tensorflow:tensorflow-android:1.2.0-preview' } Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 10

Slide 10 text

TensorFlow for Mobile (Aug 2017) Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 11

Slide 11 text

I/O 2017 Announcements Neural Networks API ‣ Android Oreo ‣ Hardware accelerated, with software fallback TensorFlow Lite ‣ Designed for low-latency on mobile/ embedded ‣ Recommended solution for new projects Image credit: TensorFlow

Slide 12

Slide 12 text

Firebase ML Kit (May 2018) ‣ Easy-to-use base APIs for ‣ Image classification, OCR, face detection ‣ Barcode scanning, landmark detection ‣ Hybrid local/cloud execution ‣ Support for custom models Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 13

Slide 13 text

Custom Models with ML Kit Image credit: burst.shopify.com

Slide 14

Slide 14 text

Custom Models with ML Kit ‣ Bring your own TF Lite model ‣ Provides API to run inference on-device ‣ Hosted models are updated automatically Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 15

Slide 15 text

Custom Models Workﬂow 1. Build custom model or retrain existing model Image credit: burst.shopify.com

Slide 16

Slide 16 text

1. Build custom model or retrain existing model 1. Use an existing model 2. Retrain an existing model 3. (Have someone) Write one from scratch Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 17

Slide 17 text

Custom Models Workﬂow 1. Build custom model or retrain existing model 2. Convert to TensorFlow Lite Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 18

Slide 18 text

2. Convert to TensorFlow Lite ‣ Use the TensorFlow Optimizing Converter ‣ Might need custom TF Lite build if operators are missing Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 19

Slide 19 text

Custom Models Workﬂow 1. Build custom model or retrain existing model 2. Convert to TensorFlow Lite 3. Bundle with app and/or host on Firebase Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 20

Slide 20 text

3. Bundle with app ‣ Include in src/main/assets ‣ Ensure model doesn't get compressed android { aaptOptions { noCompress "tflite" noCompress "lite" } } ‣ Could also download from your own server Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 21

Slide 21 text

3. Host on Firebase

Slide 22

Slide 22 text

3. Host on Firebase

Slide 23

Slide 23 text

3. Host on Firebase

Slide 24

Slide 24 text

3. Host on Firebase

Slide 25

Slide 25 text

Custom Models Workﬂow 1. Build custom model or retrain existing model 2. Convert to TensorFlow Lite 3. Bundle with app and/or host on Firebase 4. Integrate SDK into your app Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 26

Slide 26 text

4. Integrate SDK into your app ‣ Add dependency ‣ Configure sources ‣ Create interpreter ‣ Define inputs/outputs ‣ Run inference ‣ Process outputs Image credit: burst.shopify.com

Slide 27

Slide 27 text

Add the dependency dependencies { implementation 'com.google.firebase:firebase-ml-model-interpreter:16.2.2' } Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 28

Slide 28 text

Conﬁgure a bundled source FirebaseLocalModelSource localSource = new FirebaseLocalModelSource.Builder("my_local_model") .setAssetFilePath("my-custom-model.tflite") .build(); FirebaseModelManager.getInstance() .registerLocalModelSource(localSource); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Conﬁgure a downloaded source FirebaseLocalModelSource downloadedSource = new FirebaseLocalModelSource.Builder("my_local_model") .setFilePath("/path/to/downloaded/model.tflite") .build(); FirebaseModelManager.getInstance() .registerLocalModelSource(downloadedSource); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 31

Slide 31 text

Specify when hosted model is downloaded FirebaseModelDownloadConditions.Builder condBuilder = new FirebaseModelDownloadConditions.Builder() .requireWifi() .requireCharging() .requireDeviceIdle(); } FirebaseModelDownloadConditions conditions = condBuilder.build(); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 32

Slide 32 text

Specify when hosted model is downloaded FirebaseModelDownloadConditions.Builder condBuilder = new FirebaseModelDownloadConditions.Builder() .requireWifi() .requireCharging() // Android 7.0+ .requireDeviceIdle(); // Android 7.0+ } FirebaseModelDownloadConditions conditions = condBuilder.build(); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 33

Slide 33 text

Bind conditions to hosted source FirebaseCloudModelSource cloudSource = new FirebaseCloudModelSource.Builder("my-custom-model") .enableModelUpdates(true) .setInitialDownloadConditions(inConditions) .setUpdatesDownloadConditions(upConditions) .build(); FirebaseModelManager.getInstance() .registerCloudModelSource(cloudSource); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 34

Slide 34 text

Bind conditions to hosted source FirebaseCloudModelSource cloudSource = new FirebaseCloudModelSource.Builder("my-custom-model") .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build(); FirebaseModelManager.getInstance() .registerCloudModelSource(cloudSource); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 35

Slide 35 text

Create interpreter FirebaseModelOptions options = new FirebaseModelOptions.Builder() .setCloudModelName("my-custom-model") .setLocalModelName("my_local_model") .build(); FirebaseModelInterpreter interpreter = FirebaseModelInterpreter.getInstance(options); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 36

Slide 36 text

Deﬁne model input and output shapes FirebaseModelInputOutputOptions inputOutputOptions = new FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, new int[]{1, 224, 224, 3}) .setOutputFormat(0, FirebaseModelDataType.BYTE, new int[]{1, 1000}) .build(); ‣ Example for image classification model with byte input and output ‣ Use TensorBoard or Netron to check (or ask the model author) Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 37

Slide 37 text

Deﬁne inputs byte[][][][] imageBytes = new byte[1][224][224][3]; // Convert Bitmap to input… FirebaseModelInputs inputs = new FirebaseModelInputs.Builder() .add(imageBytes) .build(); Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 38

Slide 38 text

Run inference Task result = interpreter.run(inputs, inputOutputOptions) .addOnSuccessListener( new OnSuccessListener() { @Override public void onSuccess(FirebaseModelOutputs result) { // Process outputs… } }) Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 39

Slide 39 text

Process outputs FirebaseModelOutputs result = …; byte[][] output = result.getOutput(0); byte[] probabilities = output[0]; // Pick highest probability // Look up corresponding label Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 40

Slide 40 text

Things to watch for Image credit: burst.shopify.com

Slide 41

Slide 41 text

TensorFlow Lite ‣ Development on TensorFlow is very active ‣ GitHub: 110K stars, 68K forks, 12K closed issues ‣ ⚠ Broken links in documentation ‣ ⚠ Incorrect flags or commands in codelabs ‣ Check input and output shapes! ‣ Future: more operators, on-device training Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 42

Slide 42 text

ML Kit ‣ Firebase hosting only stores the model ‣ What about label files? ‣ What about versioning? ‣ SDK has no callbacks ‣ When did updated model get downloaded? ‣ What model am I running? ‣ Future: model optimization and compression ‣ Future: more domains (speech, text) Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 43

Slide 43 text

Are ML Kit custom models for you? It depends …if you can use an existing TF Lite model …if you can convert your model into TF Lite …if you want cloud hosting …if you want to get up and running now Image credit: burst.shopify.com

Slide 44

Slide 44 text

Resources Image credit: burst.shopify.com

Slide 45

Slide 45 text

Code ‣ Custom ML models with ML Kit for Firebase codelab ‣ github.com/efung/tensorflow-for-poets-2 ‣ My fork of TensorFlow for Poets codelab ‣ Added ML Kit version with live camera ‣ github.com/the-dagger/MLKitAndroid ‣ Pokémon Detector uses a custom model ‣ TensorFlow tutorials, e.g. image retraining

Slide 46

Slide 46 text

Reading ‣ ML for Android Developers • Mark Allison ‣ Who's Afraid of Machine Learning? • Britt Barak ‣ Android meets Machine Learning: From TensorFlow mobile/lite to ML Kit • Qian Jin ‣ Building “Pokédex” in Android using TensorFlow Lite and Firebase’s ML Kit • Harshit Dwivedi ‣ How to convert a custom model to TensorFlow Lite • Eric Hsiao

Slide 47

Slide 47 text

Reading ‣ Free ebook ‣ Covers TensorFlow Mobile, but has good background Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 48

Slide 48 text

Help ‣ Firebase Community Slack ‣ StackOverflow ‣ tensorflow-lite ‣ firebase-mlkit ‣ TensorFlow GitHub issues ! Using Custom Models with ML Kit • Eric Fung • Ohio DevFest 2018

Slide 49

Slide 49 text

The End ‣ Contact [email protected] • @gnufmuffin ‣ Work shopify.com/careers ‣ Blog code.gnufmuffin.com ‣ Slides speakerdeck.com/efung