Tweet
Tweet

Slide 1

Slide 1 text

Customize Your App With MLKit Victoria Gonda

Slide 2

Slide 2 text

Hello! I'm Victoria Gonda I'm an Android Engineer at Buffer and author on RayWenderlich.com You can find me on Twitter at @TTGonda

Slide 3

Slide 3 text

MLKit

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

No content

Slide 6

Slide 6 text

On Device In the Cloud

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

Barcode scanning Face detection Image labeling Landmark detection Object detection and tracking Text recognition Custom Language ID On device translation Smart reply

Slide 9

Slide 9 text

Erik Hellman - Machine Learning on mobile with MLKit Øredev 2018

Slide 10

Slide 10 text

Vision APIs

Slide 11

Slide 11 text

Object Detection and Tracking ◍ "Localize and track in real time the most prominent object in the live camera feed."

Slide 12

Slide 12 text

implementation 'com.google.firebase:firebase-ml-vision:24.0.0' implementation 'com.google.firebase:firebase-ml-vision-object- detection-model:19.0.2'

Slide 13

Slide 13 text

val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode( FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE) .enableMultipleObjects() .enableClassification() .build()

Slide 14

Slide 14 text

val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode( FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE) .enableMultipleObjects() .enableClassification() .build()

Slide 15

Slide 15 text

val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode( FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE) .enableMultipleObjects() .enableClassification() .build()

Slide 16

Slide 16 text

val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode( FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE) .enableMultipleObjects() .enableClassification() .build()

Slide 17

Slide 17 text

val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode( FirebaseVisionObjectDetectorOptions.SINGLE_IMAGE_MODE) .enableMultipleObjects() .enableClassification() .build()

Slide 18

Slide 18 text

val objectDetector = FirebaseVision.getInstance() .getOnDeviceObjectDetector(options)

Slide 19

Slide 19 text

val image = FirebaseVisionImage.fromBitmap(selectedImage)

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

objectDetector.processImage(image) .addOnSuccessListener { detectedObjects -> // Process result } .addOnFailureListener { e -> // Handle error }

Slide 22

Slide 22 text

objectDetector.processImage(image) .addOnSuccessListener { detectedObjects -> // Process result } .addOnFailureListener { e -> // Handle error }

Slide 23

Slide 23 text

objectDetector.processImage(image) .addOnSuccessListener { detectedObjects -> // Process result } .addOnFailureListener { e -> // Handle error }

Slide 24

Slide 24 text

firebaseVisionObject.boundingBox firebaseVisionObject.trackingId // null in SINGLE_IMAGE_MODE firebaseVisionObject.classificationCategory firebaseVisionObject.classificationConfidence

Slide 25

Slide 25 text

// Swift for iOS let options = VisionObjectDetectorOptions() options.detectorMode = .singleImage options.shouldEnableMultipleObjects = true options.shouldEnableClassification = true

Slide 26

Slide 26 text

let objectDetector = Vision.vision() .objectDetector(options: options) let image = VisionImage(image: uiImage)

Slide 27

Slide 27 text

objectDetector.process(image) { detectedObjects, error in guard error == nil else { // Error. return } guard let detectedObjects = detectedObjects, !detectedObjects.isEmpty else { // No objects detected. return } // Success. }

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Barcode Scanning ◍ "Scan and process barcodes."

Slide 30

Slide 30 text

Face Detection ◍ "Detect faces and facial landmarks."

Slide 31

Slide 31 text

Image Labeling ◍ "Identify objects, locations, activities, animal species, products, and more."

Slide 32

Slide 32 text

Landmark Detection ◍ "Identify popular landmarks in an image."

Slide 33

Slide 33 text

Text Recognition ◍ "Recognize and extract text from images."

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Language APIs

Slide 36

Slide 36 text

Language ID ◍ "Determine the language of a string of text with only a few words."

Slide 37

Slide 37 text

implementation 'com.google.firebase:firebase-ml-natural- language:22.0.0' implementation 'com.google.firebase:firebase-ml-natural- language-language-id-model:20.0.7'

Slide 38

Slide 38 text

val options = FirebaseLanguageIdentificationOptions .Builder() .setConfidenceThreshold(0.34f) .build()

Slide 39

Slide 39 text

val languageIdentifier = FirebaseNaturalLanguage .getInstance().getLanguageIdentification(options)

Slide 40

Slide 40 text

languageIdentifier.identifyLanguage(text) .addOnSuccessListener { languageCode -> // Use result } .addOnFailureListener { // Handle error }

Slide 41

Slide 41 text

languageIdentifier.identifyLanguage(text) .addOnSuccessListener { languageCode -> // Use result } .addOnFailureListener { // Handle error }

Slide 42

Slide 42 text

if (languageCode == "und") { // Language not confidently detected } else { // Use language code }

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

On Device Translation ◍ "Translate text between 58 languages, entirely on device."

Slide 45

Slide 45 text

implementation 'com.google.firebase:firebase-ml-natural- language:22.0.0' implementation 'com.google.firebase:firebase-ml-natural- language-translate-model:20.0.7'

Slide 46

Slide 46 text

val options = FirebaseTranslatorOptions.Builder() .setSourceLanguage(FirebaseTranslateLanguage.ES) .setTargetLanguage(FirebaseTranslateLanguage.EN) .build()

Slide 47

Slide 47 text

val options = FirebaseTranslatorOptions.Builder() .setSourceLanguage(FirebaseTranslateLanguage.ES) .setTargetLanguage(FirebaseTranslateLanguage.EN) .build()

Slide 48

Slide 48 text

val options = FirebaseTranslatorOptions.Builder() .setSourceLanguage(FirebaseTranslateLanguage.ES) .setTargetLanguage(FirebaseTranslateLanguage.EN) .build()

Slide 49

Slide 49 text

val options = FirebaseTranslatorOptions.Builder() .setSourceLanguage(FirebaseTranslateLanguage.ES) .setTargetLanguage(FirebaseTranslateLanguage.EN) .build()

Slide 50

Slide 50 text

val spanishEnglishTranslator = FirebaseNaturalLanguage .getInstance().getTranslator(options)

Slide 51

Slide 51 text

spanishEnglishTranslator.downloadModelIfNeeded() .addOnSuccessListener { // Model downloaded successfully // Okay to start translating } .addOnFailureListener { exception -> // Handle error }

Slide 52

Slide 52 text

spanishEnglishTranslator.downloadModelIfNeeded() .addOnSuccessListener { // Model downloaded successfully // Okay to start translating } .addOnFailureListener { exception -> // Handle error }

Slide 53

Slide 53 text

spanishEnglishTranslator.translate(text) .addOnSuccessListener { translatedText -> // Use translated text } .addOnFailureListener { exception -> // Handle error }

Slide 54

Slide 54 text

spanishEnglishTranslator.translate(text) .addOnSuccessListener { translatedText -> // Use translated text } .addOnFailureListener { exception -> // Handle error }

Slide 55

Slide 55 text

No content

Slide 56

Slide 56 text

Smart Reply ◍ "Generate reply suggestions in text conversations."

Slide 57

Slide 57 text

implementation 'com.google.firebase:firebase-ml-natural- language:22.0.0' implementation 'com.google.firebase:firebase-ml-natural- language-smart-reply-model:20.0.7'

Slide 58

Slide 58 text

android { aaptOptions { noCompress "tflite" } }

Slide 59

Slide 59 text

val conversation = mutableListOf() conversation.add(FirebaseTextMessage.createForLocalUser( "Hi!", System.currentTimeMillis()))

Slide 60

Slide 60 text

val conversation = mutableListOf() conversation.add(FirebaseTextMessage.createForLocalUser( "Hi!", System.currentTimeMillis()))

Slide 61

Slide 61 text

conversation.add(FirebaseTextMessage.createForRemoteUser( "It was great meeting you at Øredev!", System.currentTimeMillis(), userId)) conversation.add(FirebaseTextMessage.createForRemoteUser( "Want to keep in touch?", System.currentTimeMillis(), userId))

Slide 62

Slide 62 text

val smartReply = FirebaseNaturalLanguage.getInstance() .smartReply

Slide 63

Slide 63 text

smartReply.suggestReplies(conversation) .addOnSuccessListener { result -> if (result.status == STATUS_NOT_SUPPORTED_LANGUAGE) { // The conversation's language isn't supported } else if (result.status == STATUS_SUCCESS) { // Show suggestions } } .addOnFailureListener { // Handle error }

Slide 64

Slide 64 text

smartReply.suggestReplies(conversation) .addOnSuccessListener { result -> if (result.status == STATUS_NOT_SUPPORTED_LANGUAGE) { // The conversation's language isn't supported } else if (result.status == STATUS_SUCCESS) { // Show suggestions } } .addOnFailureListener { // Handle error }

Slide 65

Slide 65 text

smartReply.suggestReplies(conversation) .addOnSuccessListener { result -> if (result.status == STATUS_NOT_SUPPORTED_LANGUAGE) { // The conversation's language isn't supported } else if (result.status == STATUS_SUCCESS) { // Show suggestions } } .addOnFailureListener { // Handle error }

Slide 66

Slide 66 text

smartReply.suggestReplies(conversation) .addOnSuccessListener { result -> if (result.status == STATUS_NOT_SUPPORTED_LANGUAGE) { // The conversation's language isn't supported } else if (result.status == STATUS_SUCCESS) { // Show suggestions } } .addOnFailureListener { // Handle error }

Slide 67

Slide 67 text

result.suggestions.forEach { // Show suggestion }

Slide 68

Slide 68 text

No content

Slide 69

Slide 69 text

AutoML Vision Edge

Slide 70

Slide 70 text

AutoML Vision Edge ◍ "Generate custom image classification models to use on device from your own library of images."

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

implementation 'com.google.firebase:firebase-ml-vision:24.0.0' implementation 'com.google.firebase:firebase-ml-vision-automl: 18.0.2'

Slide 73

Slide 73 text

val remoteModel = FirebaseAutoMLRemoteModel .Builder("recipe_model") .build()

Slide 74

Slide 74 text

val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() FirebaseModelManager.getInstance() .download(remoteModel, conditions) .addOnCompleteListener { // Success. }

Slide 75

Slide 75 text

val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() FirebaseModelManager.getInstance() .download(remoteModel, conditions) .addOnCompleteListener { // Success. }

Slide 76

Slide 76 text

android { aaptOptions { noCompress "tflite" } }

Slide 77

Slide 77 text

val localModel = FirebaseAutoMLLocalModel.Builder() .setAssetFilePath("manifest.json") .build()

Slide 78

Slide 78 text

val options = FirebaseVisionOnDeviceAutoMLImageLabelerOptions .Builder(localModel) // or remoteModel .setConfidenceThreshold(0.5f) .build() val labeler = FirebaseVision .getInstance() .getOnDeviceAutoMLImageLabeler(options)

Slide 79

Slide 79 text

val options = FirebaseVisionOnDeviceAutoMLImageLabelerOptions .Builder(localModel) // or remoteModel .setConfidenceThreshold(0.5f) .build() val labeler = FirebaseVision .getInstance() .getOnDeviceAutoMLImageLabeler(options)

Slide 80

Slide 80 text

val image = FirebaseVisionImage.fromBitmap(selectedImage) labeler.processImage(image) .addOnSuccessListener { labels -> // Use labels } .addOnFailureListener { e -> // :( }

Slide 81

Slide 81 text

val image = FirebaseVisionImage.fromBitmap(selectedImage) labeler.processImage(image) .addOnSuccessListener { labels -> // Use labels } .addOnFailureListener { e -> // :( }

Slide 82

Slide 82 text

for (label in labels) { val text = label.text val confidence = label.confidence }

Slide 83

Slide 83 text

AutoML with Object Detection Detect object in image Apply ML model Crop image

Slide 84

Slide 84 text

Case Studies

Slide 85

Slide 85 text

Zyl

Slide 86

Slide 86 text

Lose It!

Slide 87

Slide 87 text

Slide 88

Slide 88 text

Thanks! You can find me at @TTGonda & VictoriaGonda.com