Combine CameraX and MLKit

Combine CameraX and MLKit

GDGミートアップ in 京都 #2での発表資料です。
https://gdgkyoto.connpass.com/event/163618/

## 使ったコード
- https://github.com/kwmt/DetectFaceWithCameraX

## 参考資料
- Android Jetpack: Understand the CameraX Camera-Support Library (Google I/O'19)
https://youtu.be/kuv8uK-5CLY

- What’s New with CameraX (Android Dev Summit ‘19)
https://www.youtube.com/watch?v=49lU1NtYE5g

- Recognize Text in Images with ML Kit on Android
https://firebase.google.com/docs/ml-kit/android/recognize-text?hl=en

Ce6acc3536b0e0340b5f0569d3394c9c?s=128

Yasutaka Kawamoto

February 16, 2020
Tweet

Transcript

  1. 5.

    • If you are an Android developer, You may have

    wanted to do the following: This is quite sudden, but… - Take a photo with Android camera - Analyze the photo image
  2. 6.

    • Camera API • Camera2 API • CameraX APIs currently

    available for capturing photos on Android
  3. 9.

    • Callback hell • Need to wait for camera preparation

    • Need to wait for TextureView ready • Need to wait for the Camera Capture Session to be ready • Need to wait for retrieving image data from Camera Capture Session • etc… Camera2 API is hard to use
  4. 10.

    • For some models, the camera function orientation cannot be

    specified. • Some models cannot be captured • In the first place, some models cannot be previewed • etc… Many model-dependent problems
  5. 12.

    • Current version is alpha10 • Beta version will be

    released by the end of the first quarter of 2020
 • Camera X uses Camera2 API • Works from API Level21(5.0, Lolipop) What is CameraX https://groups.google.com/a/android.com/forum/#!topic/camerax-developers/pvkt-mcz-RI
  6. 13.
  7. 16.

    Create UseCase instances // Create Preview UseCase preview = Preview.Builder()

    .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build()
  8. 17.

    Create UseCase instances // Create Preview UseCase preview = Preview.Builder()

    .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageAnalyzer UseCase imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build()
  9. 18.

    Create UseCase instances // Create Preview UseCase preview = Preview.Builder()

    .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageCapture UseCase imageCapture = ImageCapture.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageAnalyzer UseCase imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build()
  10. 19.

    Create UseCase instances // Create Preview UseCase preview = Preview.Builder()

    .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageCapture UseCase imageCapture = ImageCapture.Builder() .setTargetResolution(
 Size(viewFinder.width, viewFinder.height)) .build() // Create ImageAnalyzer UseCase imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build()
  11. 20.

    Bind UseCases to a LifecycleOwner val cameraProvider = ProcessCameraProvider .getInstance(requireContext()).get()

    cameraProvider.bindToLifecycle(
 this as LifecycleOwner, 
 cameraSelector, 
 preview, imageCapture, imageAnalyzer)
  12. 23.

    • Google is able to guarantee a high quality camera

    performance • Automated CameraX test lab • Populated with devices from various manufactures • Test hundreds of devices • Various test types • Functional, integration, end-to-end use case, performance, etc. Consistent behavior
  13. 26.

    • ML Kit beta brings Google’s machine learning expertise to

    mobile developers in a powerful and easy-to-use package. What is ML Kit
  14. 27.

    • ML Kit comes with a set of ready-to-use APIs

    for common mobile use cases: • recognizing text • detecting faces • identifying landmarks • scanning barcodes • etc… • Simply pass in data to the ML Kit library and it gives you the information you need. What is ML Kit
  15. 28.

    Code sample of recognizing text val visionImage = FirebaseVisionImage .fromMediaImage(image,

    rotation) Create a FirebaseVisionImage object from your image
  16. 29.

    Code sample of recognizing text val detector = FirebaseVision.getInstance() .onDeviceTextRecognizer

    val visionImage = FirebaseVisionImage .fromMediaImage(image, rotation) Create a FirebaseVisionImage object from your image Get an instance of FirebaseVisionTextRecognizer
  17. 30.

    Code sample of recognizing text val detector = FirebaseVision.getInstance() .onDeviceTextRecognizer

    val visionImage = FirebaseVisionImage .fromMediaImage(image, rotation) Create a FirebaseVisionImage object from your image Get an instance of FirebaseVisionTextRecognizer Finally, pass the image to the processImage method: detector.processImage(visionImage) .addOnSuccessListener { results -> // something }.addOnFailureListener { e -> // something }
  18. 33.

    Set analyzer val textAnalyzer = TextAnalyzer() // Create ImageAnalyzer UseCase

    imageAnalysis = ImageAnalysis.Builder() .setTargetResolution( Size(viewFinder.width, viewFinder.height)) .build() .also {it.setAnalyzer(mainExecutor, textAnalyzer)}
  19. 34.

    Combine CameraX and MLKit class TextAnalyzer : ImageAnalysis.Analyzer { override

    fun analyze(imageProxy: ImageProxy) { val image = imageProxy.image!! val rotation = // imageProxy.imageInfo.rotationDegrees to rotation of Firebase val visionImage = FirebaseVisionImage .fromMediaImage(image, rotation) imageProxy.close() detector.processImage(image) .addOnSuccessListener { results -> // something }.addOnFailureListener { e -> // something } } }
  20. 35.
  21. 37.

    • CameraX simplifies Camera2 • Easy machine learning with MLKit

    • Easy combination of CameraX and MLKit Conclusion
  22. 39.

    • CameraX • Android Jetpack: Understand the CameraX Camera- Support

    Library (Google I/O'19) • What’s New with CameraX (Android Dev Summit ‘19) • MLKIt • Recognize Text in Images with ML Kit on Android References
  23. 40.