Building Smarter Apps with MLKit

Slide 1

Slide 1 text

Building smarter apps with ML KIT Moyinoluwa Adeyemi @moyheen

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

STATE OF ML ON ANDROID

Slide 4

Slide 4 text

STATE OF ML ON ANDROID

Slide 5

Slide 5 text

“a new SDK that brings Google's machine learning expertise to mobile developers in a powerful, yet easy-to-use package on Firebase.”

Slide 6

Slide 6 text

Come one, come all!

Slide 7

Slide 7 text

Common mobile use cases ● Text recognition ● Face detection ● Barcode scanning

Slide 8

Slide 8 text

Common mobile use cases ● Text recognition ● Face detection ● Barcode scanning ● Image labeling ● Landmark recognition

Slide 9

Slide 9 text

Common mobile use cases - tbr ● Smart replies ● High density face contour addition

Slide 10

Slide 10 text

Source: https://firebase.google.com/docs/ml-kit/ On-device and cloud apis

Slide 11

Slide 11 text

BYOCM - bring your own custom models Source: https://proandroiddev.com/tensorflow-hands-on-with-android-2d0134cc251b

Slide 12

Slide 12 text

Firebase hosting for custom models ● HOW TO INCLUDE THE CUSTOM MODEL IN THE APP?

Slide 13

Slide 13 text

Firebase hosting for custom models ● HOW TO INCLUDE THE CUSTOM MODEL IN THE APP? ● SECURITY?

Slide 14

Slide 14 text

Firebase hosting for custom models ● HOW TO INCLUDE THE CUSTOM MODEL IN THE APP? ● SECURITY? ● HOW TO DOWNLOAD THE CUSTOM MODEL?

Slide 15

Slide 15 text

Firebase hosting for custom models ● HOW TO INCLUDE THE CUSTOM MODEL IN THE APP? ● SECURITY? ● HOW TO DOWNLOAD THE CUSTOM MODEL? ● HOW TO UPDATE THE CUSTOM MODEL?

Slide 16

Slide 16 text

BASE APIS

Slide 17

Slide 17 text

Base APIs Text recognition

Slide 18

Slide 18 text

Source: https://firebase.google.com/docs/ml-kit/recognize-text On-device and cloud apis

Slide 19

Slide 19 text

A. POP Quiz: Which of these were detected on-device? B.

Slide 20

Slide 20 text

B. POP Quiz: Which of these were detected on-device? A.

Slide 21

Slide 21 text

Base APIs FACE DETECTION

Slide 22

Slide 22 text

Euler X - up/down Euler Y - left/right Euler Z - rotated/slated Understands faces positioned at different angles https://developers.google.com/vision/face-detection-concepts

Slide 23

Slide 23 text

DETECTS LANDMARKS https://pixabay.com/en/woman-stylish-fashion-view-101542/ LEFT AND RIGHT EAR - 3, 9 LEFT AND RIGHT EYE - 4, 10 NOSE BASE- 6 LEFT AND RIGHT CHEEK - 1, 7 LEFT, RIGHT AND BOTTOM MOUTH - 5, 11, 0

Slide 24

Slide 24 text

UNDERSTANDS FACIAL EXPRESSIONS SMILING PROBABILITY: 0.006698033 LEFT EYE OPEN PROBABILITY: 0.98714304 RIGHT EYE OPEN PROBABILITY: 0.69178355 https://pixabay.com/en/woman-stylish-fashion-view-101542/

Slide 25

Slide 25 text

WORKS ON ALL SKIN COLORS

Slide 26

Slide 26 text

Base APIs Barcode Scanning

Slide 27

Slide 27 text

https://www.adazonusa.com/blog/wp-content/uploads/2016/03/1D-barcode-vs-2D-barcodes.jpg WORKS FOR 1D AND 2D BARCODES

Slide 28

Slide 28 text

DETECTS MULTIPLE BARCODES IN AN IMAGE

Slide 29

Slide 29 text

EVEN WHEN THEY ARE UPSIDE DOWN

Slide 30

Slide 30 text

Base APIs IMAGE LABELING

Slide 31

Slide 31 text

On-device and cloud apis

Slide 32

Slide 32 text

SUPPORTS DIFFERENT LABELS

Slide 33

Slide 33 text

Base APIs LANDMARK RECOGNITION

Slide 34

Slide 34 text

GETTING STARTED GENERAL STEPS

Slide 35

Slide 35 text

CONNECT TO FIREBASE

Slide 36

Slide 36 text

Add the dependency to gradle

Slide 37

Slide 37 text

Add an extra dependency to gradle for image labeling

Slide 38

Slide 38 text

GETTING STARTED On-device apis

Slide 39

Slide 39 text

Add meta-data to the manifest file

Slide 40

Slide 40 text

GETTING STARTED Cloud apis

Slide 41

Slide 41 text

Upgrade to the Blaze plan

Slide 42

Slide 42 text

ENABLE THE CLOUD VISION API

Slide 43

Slide 43 text

IMPLEMENTATION (FACE DETECTION API)

Slide 44

Slide 44 text

Implementation ● Configure the detector options ● Run the detector ● RETRIEVE THE INFORMATION

Slide 45

Slide 45 text

CONFIGURE THE DETECTOR OPTIONS FirebaseVisionFaceDetectorOptions.Builder() .setModeType(ACCURATE_MODE) .setLandmarkType(ALL_LANDMARKS) .setClassificationType(ALL_CLASSIFICATIONS) .setMinFaceSize(0.15f) .setTrackingEnabled(true) .build()

Slide 46

Slide 46 text

FirebaseVisionFaceDetectorOptions.Builder() .setModeType(ACCURATE_MODE) .setLandmarkType(ALL_LANDMARKS) .setClassificationType(ALL_CLASSIFICATIONS) .setMinFaceSize(0.15f) .setTrackingEnabled(true) .build() CONFIGURE THE DETECTOR OPTIONS

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

Slide 52

Slide 52 text

Implementation ● Configure the detector options ● Run the detector ● RETRIEVE THE INFORMATION

Slide 53

Slide 53 text

val detector = FirebaseVision.getInstance() .getVisionFaceDetector(options) RUN THE DETECTOR

Slide 54

Slide 54 text

val image = FirebaseVisionImage.fromBitmap(bitmap) RUN THE DETECTOR

Slide 55

Slide 55 text

val image = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(image) .addOnSuccessListener { } .addOnFailureListener { } RUN THE DETECTOR

Slide 56

Slide 56 text

val image = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(image) .addOnSuccessListener { } .addOnFailureListener { } RUN THE DETECTOR

Slide 57

Slide 57 text

Implementation ● Configure the detector options ● Run the detector ● RETRIEVE THE INFORMATION

Slide 58

Slide 58 text

RETRIEVE THE INFORMATION detector.detectInImage(image) .addOnSuccessListener { faces -> // Task completed successfully faces.forEach { face -> face.smilingProbability face.rightEyeOpenProbability face.getLandmark(LEFT_EAR) } }

Slide 59

Slide 59 text

detector.detectInImage(image) .addOnSuccessListener { faces -> // Task completed successfully faces.forEach { face -> face.smilingProbability face.rightEyeOpenProbability face.getLandmark(LEFT_EAR) } } RETRIEVE THE INFORMATION

Slide 60

Slide 60 text

Slide 61

Slide 61 text

detector.detectInImage(image) .addOnFailureListener { error -> // Task failed with an exception displayError(error.message) } RETRIEVE THE INFORMATION

Slide 62

Slide 62 text

detector.detectInImage(image) .addOnFailureListener { error -> // Task failed with an exception displayError(error.message) } RETRIEVE THE INFORMATION

Slide 63

Slide 63 text

CUSTOM MODELS

Slide 64

Slide 64 text

Custom Models

Slide 65

Slide 65 text

GETTING STARTED GENERAL STEPS

Slide 66

Slide 66 text

CONNECT TO FIREBASE

Slide 67

Slide 67 text

Add the dependency to gradle

Slide 68

Slide 68 text

Convert To .tflite Format With TOCO TOCO: TensorFlow LITE OPTIMIZING CONVERTER

Slide 69

Slide 69 text

GETTING STARTED CUSTOM MODELS HOSTED ON FIREBASE

Slide 70

Slide 70 text

INTERNET PERMISSION

Slide 71

Slide 71 text

Upload .tflite model to firebase

Slide 72

Slide 72 text

GETTING STARTED CUSTOM MODELS ON DEVICE

Slide 73

Slide 73 text

Bundle model with app

Slide 74

Slide 74 text

Add to gradle

Slide 75

Slide 75 text

IMPLEMENTATION

Slide 76

Slide 76 text

val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() // requires API Level 24 .requireCharging() // requires API Level 24 .requireDeviceIdle() .build() SPECIFY DOWNLOAD CONDITIONS

Slide 77

Slide 77 text

val cloudSource = FirebaseCloudModelSource.Builder("mobilenet_v1_224_quant") .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() Create a FirebaseCloudModelSource

Slide 78

Slide 78 text

private const val ASSET = "mobilenet_v1.0_224_quant.tflite" val localSource = FirebaseLocalModelSource.Builder("asset") .setFilePath("/filepath") .setAssetFilePath(ASSET) .build() AND/OR a FirebaseLocalModelSource

Slide 79

Slide 79 text

FirebaseModelManager.getInstance().apply { registerLocalModelSource(localSource) registerCloudModelSource(cloudSource) } REGISTER THE MODELS

Slide 80

Slide 80 text

val modelOptions = FirebaseModelOptions.Builder() .setCloudModelName(HOSTED_MODEL) .setLocalModelName(LOCAL_MODEL) .build() modelInterpreter = FirebaseModelInterpreter.getInstance(modelOptions) Get instance of FirebaseModelInterpreter

Slide 81

Slide 81 text

val input = intArrayOf(1, 224, 224, 3) val output = intArrayOf(1, labelList.size) FirebaseModelInputOutputOptions.Builder() .setInputFormat(0, FirebaseModelDataType.BYTE, input) .setOutputFormat(0, FirebaseModelDataType.BYTE, output) .build() SPECIFY INPUT AND OUTPUT FORMAT

Slide 82

Slide 82 text

val imageData = convertBitmapToByteBuffer(BitmapFactory.decodeResource(resources, R.drawable.tennis)) val inputs = FirebaseModelInputs.Builder() .add(imageData) .build() Create the input

Slide 83

Slide 83 text

modelInterpreter ?.run(inputs, options) ?.addOnSuccessListener { result -> val labelProbArray = result.getOutput>(0) // do something with labelProbArray } ?.addOnFailureListener { error -> // display error } Run the interpreter

Slide 84

Slide 84 text

Run the interpreter

Slide 85

Slide 85 text

ML Kit makes it really easy for Android Developers to build smarter apps. Summary

Slide 86

Slide 86 text

- THIS TALK BY YUFENG GUO https://www.youtube.com/watch?v=EnFyneRScQ8 - All the google codelabs on Tensorflow, TensorFlowLite and MLKit - ML Kit official documentation - Sketches - https://www.thedoodlelibrary.com/ Resources

Slide 87

Slide 87 text

THANK YOU! Moyinoluwa Adeyemi @moyheen