Share
Embed
new ML Kit feature
Shibuya.apk #34 - Report from Google I/O 2019
https://shibuya-apk.connpass.com/event/128668/
Yuki Anzai
May 17, 2019
Transcript
33 slides
-
new ML Kit feature Yuki Anzai (@yanzm) Shibuya.apk #34 -
Report from Google I/O 2019 -
Yuki Anzai • Android App Developer (2009~) • CEO of
uPhyca Inc. (2011~) • Google Developer Expert for Android • Organizer of GTUG Girls and droid girls • Twitter : @yanzm -
ML Kit ͱ • Firebase ͷػೳͷҰͭ • ػցֶशΛར༻͢ΔػೳΛΞϓϦʹ؆୯ʹΈࠐΉͨΊͷϞόΠϧ SDK •
ݱࡏ β • iOS ͱ Android Ͱ͑Δ • https:// fi rebase.google.com/docs/ml-kit/ -
on-device vs Cloud • on-device API • ϩʔΧϧͰಈ࡞ɺ͍ • Firebase
͕ػցֶशͷϞσϧΛ͋Β͔͡Ίμϯϩʔυͯ͘͠ΕΔ • Cloud API • αʔόʔͰॲཧɺߴػೳ • ωοτϫʔΫଓ͕ඞཁ -
͍··Ͱͷػೳ on-device Cloud Text recognition : ςΩετೝࣝ O O Face
detection : إݕग़ O - Barcode scanning : όʔίʔυεΩϟϯ O - Image labeling : ը૾ͷϥϕϧ͚ O O Landmark recognition : ϥϯυϚʔΫೝࣝ - O Custom model inference : ΧελϜϞσϧਪ O - -
৽͘͠Ճ͞Εͨػೳ • Object Detection and Tracking • on-device Translation •
ML Kit: Machine Learning for Mobile with Firebase (Google I/O'19) • https://www.youtube.com/watch?v=QwHD36bhXZA -
on-device Translation API • 59 ݴޠؒͷ༁͕Մೳ • https:// fi rebase.google.com/docs/ml-kit/translation-language-
support • Google Translate ͷ of fl ine mode ͱಉ͡Ϟσϧ • શʹ device ্Ͱಈ࡞ = of fl ine Ͱಈ͘ • ແྉ -
Language Pack • Each language pack : 25~35 MB •
Language packs ಈతʹμϯϩʔυ͞ΕΔ • 59 language → ~3000 pairs • English Λதؒݴޠͱͯ͠ར༻ • ྫ) Dutch → English → Chinese -
Limitations • intended for casual and simple translations • ༁ͷ࣭ݴޠʹґଘ͢Δ
• ΑΓߴ͍࣮͕ඞཁͳͱ͖ Cloud Translation API Λࢼ͢ • https://cloud.google.com/translate/ -
Usage guidelines • Usage Guidelines for ML Kit On-device Translation
ΛಡΉ͜ͱ • https:// fi rebase.google.com/docs/ml-kit/translation-terms -
Providing feedback • ෆదͳ༁Λݟ͚ͭͨ࣌ Firebase support ͔Β feedback ΛૹΔ •
https:// fi rebase.google.com/support -
val options = FirebaseTranslatorOptions.Builder() .setSourceLanguage(FirebaseTranslateLanguage.EN) .setTargetLanguage(FirebaseTranslateLanguage.JA) .build() val translator =
FirebaseNaturalLanguage.getInstance().getTranslator(options) ࣮ࡍʹ translate ͢ΔલʹϞσϧ͕μϯϩʔυ͞Ε͍ͯΔ͔֬ೝ͢Δ val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() translator.downloadModelIfNeeded(conditions) .addOnSuccessListener { } .addOnFailureListener { } -
Explicitly manage translation models • ML Kit's translation model management
API • ࣄલʹϞσϧΛμϯϩʔυ • ෆཁͳϞσϧΛআ -
val modelManager = FirebaseTranslateModelManager.getInstance() // σόΠεʹอଘ͞Ε͍ͯΔ༁ϞσϧΛऔಘ͢Δ modelManager.getAvailableModels(FirebaseApp.getInstance()) .addOnSuccessListener { models:
Set<FirebaseTranslateRemoteModel> -> // ... } .addOnFailureListener { // Error. } -
val modelManager = FirebaseTranslateModelManager.getInstance() // ຊޠϞσϧΛʢσόΠεʹอଘ͞Ε͍ͯΕʣআ͢Δ val jaModel = FirebaseTranslateRemoteModel.Builder(FirebaseTranslateLanguage.JA)
.build() modelManager.deleteDownloadedModel(jaModel) .addOnSuccessListener { // Model deleted. } .addOnFailureListener { // Error. } -
val modelManager = FirebaseTranslateModelManager.getInstance() // ຊޠϞσϧΛ໌ࣔతʹμϯϩʔυ͢Δ val conditions = FirebaseModelDownloadConditions.Builder()
.requireWifi() .build() val jaModel = FirebaseTranslateRemoteModel.Builder(FirebaseTranslateLanguage.JA) .setDownloadConditions(conditions) .build() modelManager.downloadRemoteModelIfNeeded(jaModel) .addOnSuccessListener { // Model downloaded. } .addOnFailureListener { // Error. } -
Object Detection and Tracking • ը૾͔ΒΦϒδΣΫτΛݕग़ • ୯Ұݕग़ or ෳݕग़
• ΧςΰϦྨʢ6छྨʣ͋Γ or ͳ͠ • on-device • ແྉ -
Codelab • Detect objects in images with ML Kit for
Firebase: Android • https://codelabs.developers.google.com/codelabs/mlkit-android- odt/index.html • ੩ࢭը͔ΒΦϒδΣΫτΛݕग़ -
Stream or Single Image • STREAM_MODE • low latency (࠷ॳͷճͷݺͼग़͠Ͱෆશͳ݁Ռ͕ฦΔ͜ͱ͕͋Δ)
• tracking ID • ಈըϓϨϏϡʔ͚ • SINGLE_IMAGE_MODE • STREAM_MODE ΑΓ latency ͕ߴ͘ͳΓ͕ͪ • ੩ࢭը͚ -
ΧςΰϦ • home goods • fashion goods • food •
plants • places • unknown -
val options = FirebaseVisionObjectDetectorOptions.Builder() .setDetectorMode(FirebaseVisionObjectDetectorOptions.STREAM_MODE) // or SINGLE_IMAGE_MODE .enableClassification() //
Optional : ෳݕग़ .enableMultipleObjects() // Optional : ΧςΰϦ .build() val detector = FirebaseVision.getInstance().getOnDeviceObjectDetector(options) -
val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation) val image = FirebaseVisionImage.fromBitmap(bitmap) detector.processImage(image)
.addOnSuccessListener { results -> for (obj in results) { val entityId: String = obj.entityId val box: Rect = obj.boundingBox val category: Int = obj.classificationCategory val confidence: Float? = obj.classificationConfidence val trackingId: Int? = obj.trackingId ... } } .addOnFailureListener { ... } -
ಈըϓϨϏϡʔͰ࠷ྑͷϑϨʔϜϨʔτΛୡ͢Δʹ • ෳݕग़ΛΘͳ͍ • ෆཁͳ߹ΧςΰϦྨΛແޮʹ͢Δ • detector ͷ࣮ߦதʹར༻Մೳʹͳͬͨ৽͍͠ϑϨʔϜυϩοϓ͢Δ •
Camera 2 API Λ༻͍ͯ͠Δ߹ ImageFormat.YUV_420_888ܗࣜ Ͱɺݹ͍ Camera API Λ༻͍ͯ͠Δ߹ ImageFormat.NV21ܗࣜͰ ը૾ΛΩϟϓνϟ͢Δ -
Patterns for machine learning-powered features • Material Design • https://material.io/collections/machine-learning/
• αϯϓϧΞϓϦ • https://github.com/ fi rebase/mlkit-material-android -
Custom Image Classi fi er • σʔληοτऩूɺϥϕϧ͚ɺֶशʢML Kit ͷ AutoML
Vision EdgeʣɺϞσϧͷςετ͕Ͱ͖Δศར Flutter ΞϓϦ • https://github.com/ fi rebase/mlkit-custom-image-classi fi er