new ML Kit feature Yuki Anzai (@yanzm) Shibuya.apk #34 - Report from Google I/O 2019 

Yuki Anzai • Android App Developer (2009~) • CEO of uPhyca Inc. (2011~) • Google Developer Expert for Android • Organizer of GTUG Girls and droid girls • Twitter : @yanzm 

ML Kit ͱ͸ • Firebase ͷػೳͷҰͭ • ػցֶशΛར༻͢ΔػೳΛΞϓϦʹ؆୯ʹ૊ΈࠐΉͨΊͷϞόΠϧ SDK • ݱࡏ͸ β • iOS ͱ Android Ͱ࢖͑Δ • https:// fi rebase.google.com/docs/ml-kit/ 

on-device vs Cloud • on-device API • ϩʔΧϧͰಈ࡞ɺ଎͍ • Firebase ͕ػցֶशͷϞσϧΛ͋Β͔͡Ίμ΢ϯϩʔυͯ͘͠ΕΔ • Cloud API • αʔόʔͰॲཧɺߴػೳ • ωοτϫʔΫ઀ଓ͕ඞཁ 

͍··Ͱͷػೳ on-device Cloud Text recognition : ςΩετೝࣝ O O Face detection : إݕग़ O - Barcode scanning : όʔίʔυεΩϟϯ O - Image labeling : ը૾ͷϥϕϧ෇͚ O O Landmark recognition : ϥϯυϚʔΫೝࣝ - O Custom model inference : ΧελϜϞσϧਪ࿦ O - 

৽͘͠௥Ճ͞Εͨػೳ • Object Detection and Tracking • on-device Translation • ML Kit: Machine Learning for Mobile with Firebase (Google I/O'19) • https://www.youtube.com/watch?v=QwHD36bhXZA 

on-device Translation API • 59 ݴޠؒͷ຋༁͕Մೳ • https:// fi rebase.google.com/docs/ml-kit/translation-language- support • Google Translate ͷ of fl ine mode ͱಉ͡Ϟσϧ • ׬શʹ device ্Ͱಈ࡞ = of fl ine Ͱಈ͘ • ແྉ 

Language Pack • Each language pack : 25~35 MB • Language packs ͸ಈతʹμ΢ϯϩʔυ͞ΕΔ • 59 language → ~3000 pairs • English Λதؒݴޠͱͯ͠ར༻ • ྫ) Dutch → English → Chinese 

Limitations • intended for casual and simple translations • ຋༁ͷ࣭͸ݴޠʹґଘ͢Δ • ΑΓߴ͍஧࣮౓͕ඞཁͳͱ͖͸ Cloud Translation API Λࢼ͢ • https://cloud.google.com/translate/ 

Usage guidelines • Usage Guidelines for ML Kit On-device Translation ΛಡΉ͜ͱ • https:// fi rebase.google.com/docs/ml-kit/translation-terms 

Providing feedback • ෆద੾ͳ຋༁Λݟ͚ͭͨ࣌͸ Firebase support ͔Β feedback ΛૹΔ • https:// fi rebase.google.com/support 

Setup dependencies { ... implementation "com.google.firebase:firebase-ml-natural-language:19.0.0" implementation "com.google.firebase:firebase-ml-natural-language-translate-model:19.0.0" } 

val options = FirebaseTranslatorOptions.Builder() .setSourceLanguage(FirebaseTranslateLanguage.EN) .setTargetLanguage(FirebaseTranslateLanguage.JA) .build() val translator = FirebaseNaturalLanguage.getInstance().getTranslator(options) ࣮ࡍʹ translate ͢ΔલʹϞσϧ͕μ΢ϯϩʔυ͞Ε͍ͯΔ͔֬ೝ͢Δ val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() translator.downloadModelIfNeeded(conditions) .addOnSuccessListener { } .addOnFailureListener { } 

enJaTranslator.translate(text) .addOnSuccessListener { } .addOnFailureListener { } 

Explicitly manage translation models • ML Kit's translation model management API • ࣄલʹϞσϧΛμ΢ϯϩʔυ • ෆཁͳϞσϧΛ࡟আ 

val modelManager = FirebaseTranslateModelManager.getInstance() // σόΠεʹอଘ͞Ε͍ͯΔ຋༁ϞσϧΛऔಘ͢Δ modelManager.getAvailableModels(FirebaseApp.getInstance()) .addOnSuccessListener { models: Set -> // ... } .addOnFailureListener { // Error. } 

val modelManager = FirebaseTranslateModelManager.getInstance() // ೔ຊޠϞσϧΛʢσόΠεʹอଘ͞Ε͍ͯΕ͹ʣ࡟আ͢Δ val jaModel = FirebaseTranslateRemoteModel.Builder(FirebaseTranslateLanguage.JA) .build() modelManager.deleteDownloadedModel(jaModel) .addOnSuccessListener { // Model deleted. } .addOnFailureListener { // Error. } 

val modelManager = FirebaseTranslateModelManager.getInstance() // ೔ຊޠϞσϧΛ໌ࣔతʹμ΢ϯϩʔυ͢Δ val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() val jaModel = FirebaseTranslateRemoteModel.Builder(FirebaseTranslateLanguage.JA) .setDownloadConditions(conditions) .build() modelManager.downloadRemoteModelIfNeeded(jaModel) .addOnSuccessListener { // Model downloaded. } .addOnFailureListener { // Error. } 

Object Detection and Tracking • ը૾͔ΒΦϒδΣΫτΛݕग़ • ୯Ұݕग़ or ෳ਺ݕग़ • ΧςΰϦ෼ྨʢ6छྨʣ͋Γ or ͳ͠ • on-device • ແྉ 

No content

Codelab • Detect objects in images with ML Kit for Firebase: Android • https://codelabs.developers.google.com/codelabs/mlkit-android- odt/index.html • ੩ࢭը͔ΒΦϒδΣΫτΛݕग़ 

Stream or Single Image • STREAM_MODE • low latency (࠷ॳͷ਺ճͷݺͼग़͠Ͱෆ׬શͳ݁Ռ͕ฦΔ͜ͱ͕͋Δ) • tracking ID • ಈըϓϨϏϡʔ޲͚ • SINGLE_IMAGE_MODE • STREAM_MODE ΑΓ͸ latency ͕ߴ͘ͳΓ͕ͪ • ੩ࢭը޲͚ 

ෳ਺ݕग़ • ࠷େ5ݸ·Ͱݕग़ 

ΧςΰϦ • home goods • fashion goods • food • plants • places • unknown 

Setup dependencies { ... implementation "com.google.firebase:firebase-ml-vision:20.0.0" implementation "com.google.firebase:firebase-ml-vision-object-detection-model:16.0.0" } 

val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode(FirebaseVisionObjectDetectorOptions.STREAM_MODE) // or SINGLE_IMAGE_MODE .enableClassification() // Optional : ෳ਺ݕग़ .enableMultipleObjects() // Optional : ΧςΰϦ .build() val detector = FirebaseVision.getInstance().getOnDeviceObjectDetector(options) 

val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation) val image = FirebaseVisionImage.fromBitmap(bitmap) detector.processImage(image) .addOnSuccessListener { results -> for (obj in results) { val entityId: String = obj.entityId val box: Rect = obj.boundingBox val category: Int = obj.classificationCategory val confidence: Float? = obj.classificationConfidence val trackingId: Int? = obj.trackingId ... } } .addOnFailureListener { ... } 

Improving usability and performance • ࢹ֮తಛ௃͕গͳ͍ΦϒδΣΫτͩͱɺΦϒδΣΫτ͕ը૾ͷେ෦෼Λ઎ Ίͳ͍ͱݕग़͞Εͳ͍͜ͱ͕͋ΔͷͰɺ͏·͘ݕग़Ͱ͖ΔΑ͏ʹϢʔβʔ ʹద੾ͳΨΠμϯεΛఏڙ͢ΔͱΑ͍ • ΧςΰϦ෼ྨΛ༗ޮʹͯ͠ɺಛఆͷΧςΰϦͷΦϒδΣΫτ͚ͩର৅ʹ͢ ΔΑ͏ͳϢʔεέʔεͰ͸ɺunknown ʹ഑ྀ͢Δ 

ಈըϓϨϏϡʔͰ࠷ྑͷϑϨʔϜϨʔτΛୡ੒͢Δʹ ͸ • ෳ਺ݕग़Λ࢖Θͳ͍ • ෆཁͳ৔߹͸ΧςΰϦ෼ྨΛແޮʹ͢Δ • detector ͷ࣮ߦதʹར༻Մೳʹͳͬͨ৽͍͠ϑϨʔϜ͸υϩοϓ͢Δ • Camera 2 API Λ࢖༻͍ͯ͠Δ৔߹͸ ImageFormat.YUV_420_888ܗࣜ Ͱɺݹ͍ Camera API Λ࢖༻͍ͯ͠Δ৔߹͸ ImageFormat.NV21ܗࣜͰ ը૾ΛΩϟϓνϟ͢Δ 

Patterns for machine learning-powered features • Material Design • https://material.io/collections/machine-learning/ • αϯϓϧΞϓϦ • https://github.com/ fi rebase/mlkit-material-android 

Custom Image Classi fi er • σʔληοτऩूɺϥϕϧ෇͚ɺֶशʢML Kit ͷ AutoML Vision EdgeʣɺϞσϧͷςετ͕Ͱ͖Δศར Flutter ΞϓϦ • https://github.com/ fi rebase/mlkit-custom-image-classi fi er 

CameraX Ͱ ObjetDetection ͍ͨ͠ • https://github.com/yanzm/CameraXObjectDetection 

Thank you