Combine CameraX and MLKit

GDGミートアップ in 京都 #2での発表資料です。
https://gdgkyoto.connpass.com/event/163618/

## 使ったコード
- https://github.com/kwmt/DetectFaceWithCameraX

## 参考資料
- Android Jetpack: Understand the CameraX Camera-Support Library (Google I/O'19)
https://youtu.be/kuv8uK-5CLY

- What’s New with CameraX (Android Dev Summit ‘19)
https://www.youtube.com/watch?v=49lU1NtYE5g

- Recognize Text in Images with ML Kit on Android
https://firebase.google.com/docs/ml-kit/android/recognize-text?hl=en

Transcript

1. Combine CameraX and MLKit GDGϛʔτΞοϓ in ژ౎ #2 2020/02/16
2. • Yasutaka Kawamoto • Android and iOS engineer
 @tech vein Inc. • kwmt • kwmt27 Who am I
3. • Introduce CameraX Library • Introduce MLKit Library • Combine CameraX and MLKit • Conclusion Agenda
4. Introduce CameraX Library
5. • If you are an Android developer, You may have wanted to do the following: This is quite sudden, but… - Take a photo with Android camera - Analyze the photo image
6. • Camera API • Camera2 API • CameraX APIs currently available for capturing photos on Android
7. • API Level 1ʙ Available • API Level21(5.0, Lolipop)ʙ Deprecated Camera API
8. • API Level21(5.0, Lolipop)ʙ Available Camera2 API
9. • Callback hell • Need to wait for camera preparation • Need to wait for TextureView ready • Need to wait for the Camera Capture Session to be ready • Need to wait for retrieving image data from Camera Capture Session • etc… Camera2 API is hard to use
10. • For some models, the camera function orientation cannot be specified. • Some models cannot be captured • In the first place, some models cannot be previewed • etc… Many model-dependent problems
11. CameraX https://www.youtube.com/watch?v=kuv8uK-5CLY
12. • Current version is alpha10 • Beta version will be released by the end of the first quarter of 2020
 • Camera X uses Camera2 API • Works from API Level21(5.0, Lolipop) What is CameraX https://groups.google.com/a/android.com/forum/#!topic/camerax-developers/pvkt-mcz-RI
13. • Easy to use the difficulties of Camera2 • Provide a UseCase based API Easy to use
14. https://youtu.be/kuv8uK-5CLY?t=267
15. https://youtu.be/kuv8uK-5CLY?t=267
16. Create UseCase instances // Create Preview UseCase preview = Preview.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build()
17. Create UseCase instances // Create Preview UseCase preview = Preview.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageAnalyzer UseCase imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build()
18. Create UseCase instances // Create Preview UseCase preview = Preview.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageCapture UseCase imageCapture = ImageCapture.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageAnalyzer UseCase imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build()
19. Create UseCase instances // Create Preview UseCase preview = Preview.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageCapture UseCase imageCapture = ImageCapture.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageAnalyzer UseCase imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build()
20. Bind UseCases to a LifecycleOwner val cameraProvider = ProcessCameraProvider .getInstance(requireContext()).get() cameraProvider.bindToLifecycle(
 this as LifecycleOwner, 
 cameraSelector, 
 preview, imageCapture, imageAnalyzer)
21. How to take a picture imageCapture.takePicture( photoFile, metadata, mainExecutor, imageSavedListener)
22. ☺ It’s very simple
23. • Google is able to guarantee a high quality camera performance • Automated CameraX test lab • Populated with devices from various manufactures • Test hundreds of devices • Various test types • Functional, integration, end-to-end use case, performance, etc. Consistent behavior
24. ☺ It’s feel relieved
25. Introduce MLKit Library
26. • ML Kit beta brings Google’s machine learning expertise to mobile developers in a powerful and easy-to-use package. What is ML Kit
27. • ML Kit comes with a set of ready-to-use APIs for common mobile use cases: • recognizing text • detecting faces • identifying landmarks • scanning barcodes • etc… • Simply pass in data to the ML Kit library and it gives you the information you need. What is ML Kit
28. Code sample of recognizing text val visionImage = FirebaseVisionImage .fromMediaImage(image, rotation) Create a FirebaseVisionImage object from your image
29. Code sample of recognizing text val detector = FirebaseVision.getInstance() .onDeviceTextRecognizer val visionImage = FirebaseVisionImage .fromMediaImage(image, rotation) Create a FirebaseVisionImage object from your image Get an instance of FirebaseVisionTextRecognizer
30. Code sample of recognizing text val detector = FirebaseVision.getInstance() .onDeviceTextRecognizer val visionImage = FirebaseVisionImage .fromMediaImage(image, rotation) Create a FirebaseVisionImage object from your image Get an instance of FirebaseVisionTextRecognizer Finally, pass the image to the processImage method: detector.processImage(visionImage) .addOnSuccessListener { results -> // something }.addOnFailureListener { e -> // something }
31. Combine CameraX and MLKit
32. Implements ImageAnalysis.Analyzer class TextAnalyzer : ImageAnalysis.Analyzer { override fun analyze(imageProxy: ImageProxy) { } }
33. Set analyzer val textAnalyzer = TextAnalyzer() // Create ImageAnalyzer UseCase imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build() .also {it.setAnalyzer(mainExecutor, textAnalyzer)}
34. Combine CameraX and MLKit class TextAnalyzer : ImageAnalysis.Analyzer { override fun analyze(imageProxy: ImageProxy) { val image = imageProxy.image!! val rotation = // imageProxy.imageInfo.rotationDegrees to rotation of Firebase val visionImage = FirebaseVisionImage .fromMediaImage(image, rotation) imageProxy.close() detector.processImage(image) .addOnSuccessListener { results -> // something }.addOnFailureListener { e -> // something } } }
36. Conclusion
37. • CameraX simplifies Camera2 • Easy machine learning with MLKit • Easy combination of CameraX and MLKit Conclusion
38. • https://github.com/kwmt/DetectFaceWithCameraX Source used in this slide
39. • CameraX • Android Jetpack: Understand the CameraX Camera- Support Library (Google I/O'19) • What’s New with CameraX (Android Dev Summit ‘19) • MLKIt • Recognize Text in Images with ML Kit on Android References
40. Organize Androidݚڀ&ൃදձ #arap_osaka • ୈ2ճ 2019/10/26 2໊ • ୈ3ճ 2019/11/30 7໊ • ୈ1ճ 2019/9/28 5໊ • ୈ4ճ 2020/1/25 5໊
41. Thank you @kwmt27