Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Processing camera input on Android

Processing camera input on Android

Processing and analysing images from the camera in an Android application has always been a complicated thing to solve. Despite Google making a big overhaul of the camera APIs with camera2, developers still struggled with even the simplest tasks such as scanning QR-codes or doing text detection.

At Google IO 2019, we got a new library from Google to make our work easier. The new library is called CameraX and significantly simplifies the common tasks we have when working with the camera on Android. In this session, we will learn how to use the CameraX API for any kind of processing and computer vision. After this session, you’ll feel much more confident to implement the camera processing the way you wanted, without having to resort to heavy libraries that rely on the legacy APIs.

2307a37297162f815342545a2068b2f1?s=128

Erik Hellman

October 03, 2019
Tweet

Transcript

  1. PROCESSING CAMERA INPUT ON ANDROID @ERIKHELLMAN SPEAKERDECK.COM/ERIKHELLMAN

  2. AGENDA EAN: 0121 7155 Name: Eggs, 6-pack

  3. ANDROID CAMERA APIS > android.hardware.Camera (Legacy) > android.hardware.camera2 (Camera 2)

    > androidx.camera (CameraX)
  4. ANDROID CAMERA APIS > android.hardware.Camera (Legacy) > android.hardware.camera2 (Camera 2)

    > androidx.camera (CameraX)
  5. COMPUTER VISION ON ANDROID > ML Kit > ZXing >

    OpenCV > OpenGL ES > RenderScript > Play Services Vision API
  6. DEPRECATED APIS > ML Kit > ZXing > OpenCV >

    OpenGL ES > RenderScript > Play Services Vision API
  7. VIABLE CANDIDATES > ML Kit > OpenCV > OpenGL ES

    > RenderScript
  8. EXCLUDED DUE TO COMPLEXITY > ML Kit > OpenCV >

    OpenGL ES > RenderScript
  9. COMPUTER VISION ON ANDROID: CAMERAX + ML KIT1 1 Also

    works on iOS!
  10. GET STARTED QUICKLY!

  11. GRADLE DEPENDENCIES dependencies { // CameraX Core implementation "androidx.camera:camera-core:$camerax_version" //

    If you want to use Camera2 extensions implementation "androidx.camera:camera-camera2:$camerax_version" // Add Firebase ML Kit for easy computer vision functions! implementation "com.google.firebase:firebase-ml-vision:$firebaseVersion" }
  12. SETUP CAMERA PREVIEW val previewConfig = PreviewConfig.Builder() .setLensFacing(CameraX.LensFacing.BACK) .build() val

    preview = Preview(previewConfig) val viewFinder = findViewById<TextureView>(R.id.viewFinder) preview.setOnPreviewOutputUpdateListener { previewOutput -> viewFinder.surfaceTexture = previewOutput.surfaceTeexture } CameraX.bindToLifecycle(this, preview)
  13. SETUP ML KIT FOR BARCODE DETECTION val options = FirebaseVisionBarcodeDetectorOptions.Builder()

    .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options)
  14. SETUP CAMERAX IMAGE ANALYZER val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE) .setTargetResolution(Size(1024,

    768)) .build() val imageAnalysis = ImageAnalysis(imageAnalysisConfig) imageAnalysis.setAnalyzer { image, rotationDegrees -> // TODO Perform image analysis... } CameraX.bindToLifecycle(this, imageAnalysis)
  15. ANALYSE IMAGES WITH ML KIT imageAnalysis.setAnalyzer { image, rotationDegrees ->

    val imageRotation = degreesToFirebaseRotation(rotationDegrees) image?.image?.let { val visionImage = FirebaseVisionImage.fromMediaImage(it, imageRotation) detector.detectInImage(visionImage) .addOnSuccessListener { barcodes -> displayDetectedBarcode(barcodes) } } }
  16. IT WORKS, BUT...

  17. VIEW FINDER

  18. BAD AND GOOD VIEW FINDER

  19. VIEW FINDER LAYOUT <TextureView android:id="@+id/viewFinder" android:layout_width="0dp" android:layout_height="0dp" app:layout_constraintDimensionRatio="16:9" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintHorizontal_bias="0.0"

    app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent" />
  20. VIEW FINDER SKEWED (16:9)

  21. VIEW FINDER SKEWED (4:3)

  22. VIEW FINDER CORRECT (4:3)

  23. WEIRD, BUT ALSO CORRECT

  24. GOOGLE CAMERAX SAMPLES (SIMPLIFIED) val matrix = Matrix() val centerX

    = viewFinderSize.width / 2f val centerY = viewFinderSize.height / 2f val bufferRation = previewSize.height / previewSize.width.toFloat() val scaledHeight = viewFiderSize.width val scaledWidth = Math.roung(viewFinderSize.width * bufferRatio) val xScale = scaledWidth / viewFinderSize.width.toFloat() val yScale = scaledHight / viewFinderSize.height.toFloat() matrix.preScale(xScale, yScale, centerX, centerY)
  25. CAMERAX VIEW FINDER

  26. SCALE AND CROP! (SIMPLIFIED) val matrix = Matrix() val centerX

    = viewFinderSize.width / 2f val centerY = viewFinderSize.height / 2f val previewRatio = previewSize.width / previewSize.height.toFloat() val viewFinderRatio = viewFinderSize.width / viewFinderSize.height.toFloat() // Assume view finder is wider than its height matrix.postScale(1.0f, viewFinderRatio * previewRatio, centerX, centerY)
  27. None
  28. TEXTUREVIEW TRANSFORMS ON ANDROID WWW.HELLSOFT.SE/TEXTUREVIEW-TRANSFORMS-ON-ANDROID/

  29. YUV COLOR ENCODING

  30. None
  31. None
  32. None
  33. −.4 −.3 −.2 −.1 +.1 +.2 +.3 +.4 −.1 +.1

    +.2 +.3 +.4 −.2 −.3 −.4 V U
  34. YUV ENCODING

  35. YUV AND ANDROID CAMERA FRAMES imageAnalysis.setAnalyzer { image, rotationDegrees ->

    // image.format is always ImageFormat.YUV_420_888 }
  36. CONVERT YUV TO RGB (FROM WIKIPEDIA) void YUVImage::yuv2rgb(uint8_t yValue, uint8_t

    uValue, uint8_t vValue, uint8_t *r, uint8_t *g, uint8_t *b) const { int rTmp = yValue + (1.370705 * (vValue-128)); int gTmp = yValue - (0.698001 * (vValue-128)) - (0.337633 * (uValue-128)); int bTmp = yValue + (1.732446 * (uValue-128)); *r = clamp(rTmp, 0, 255); *g = clamp(gTmp, 0, 255); *b = clamp(bTmp, 0, 255); }
  37. GRAYSCALE TO THE RESCUE!

  38. EXTRACT GRAYSCALE PIXELS fun extractGrayscaleFromImage(image: Image): Pair<ByteBuffer, Size> { val

    size = Size(image.width, image.height) val yBuffer = image.planes[0].buffer val grayPixels = ByteBuffer.allocate(yBuffer.capacity()); grayPixels.put(yBuffer) grayPixels.flip() return grayPixels to size }
  39. CAMERAX GOTCHAS!

  40. CAMERAX ANALYZER MUST BE SYNCHRONOUS!2 imageAnalysis.setAnalyzer { image, rotationDegrees ->

    // This will crash - analysis must be synchronous! // See https://issuetracker.google.com/issues/139207716 GlobalScope.launch { analyzeImage(image); } } 2 Except when converting to FirebaseVisionImage!
  41. CAMERAX ANALYZER CAN'T USE IMAGEWRITER! val imageWriter = ImageWriter(renderScriptSurface, 1)

    imageAnalysis.setAnalyzer { image, rotationDegrees -> // This will crash - CameraX will call Image.close()! // See https://issuetracker.google.com/issues/139207716 val mediaImage = image?.image if (mediaImage != null) { imageWriter.queueInputImage(image.image) } }
  42. ENABLE TORCH (BACK CAMERA) val previewConfig = PreviewConfig.Builder() .setLensFacing(CameraX.LensFacing.BACK) .build()

    val preview = Preview(previewConfig) preview.enableTorch(true)
  43. ANALYZE EVERY IMAGE val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_NEXT_IMAGE) .setImageQueueDepth(5) .setTargetResolution(Size(1024,

    768)) .build()
  44. ANALYZE IMAGES IN PARALLEL // Thread pool with 5 threads

    val executor = Executors.newFixedThreadPool(5) val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_NEXT_IMAGE) .setImageQueueDepth(5) .setTargetResolution(Size(1024, 768)) .build() imageAnalysis.setAnalyzer { image, rotationDegrees -> // Analysis can run in parallel } CameraX.bindToLifecycle(this, imageAnalysis)
  45. ANALYZE LATEST IMAGE (RECOMMENDED!) val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE) .setImageQueueDepth(5)

    .setTargetResolution(Size(1024, 768)) .build()
  46. CAMERAX ANALYSIS API IS TARGET FOR CHANGE!

  47. ML KIT

  48. FIREBASE ML KIT

  49. AUTO-DOWNLOAD REQUIRED ML MODELS (OPTIONAL) <application ...> ... <meta-data android:name="com.google.firebase.ml.vision.DEPENDENCIES"

    android:value="ocr" /> <!-- To use multiple models: android:value="ocr,model2,model3" --> </application>
  50. CONVERT ROTATION DEGREES TO FIREBASE ROTATION fun degreesToFirebaseRotation(degrees: Int): Int

    = when(degrees) { 0 -> FirebaseVisionImageMetadata.ROTATION_0 90 -> FirebaseVisionImageMetadata.ROTATION_90 180 -> FirebaseVisionImageMetadata.ROTATION_180 270 -> FirebaseVisionImageMetadata.ROTATION_270 else -> throw Exception("Rotation must be 0, 90, 180, or 270.") }
  51. TEXT RECOGNITION val detector = FirebaseVision.getInstance().onDeviceTextRecognizer val image = FirebaseVisionImage.fromMediaImage(mediaImage,

    rotation) val result = detector.processImage(image) .addOnSuccessListener { resultText -> // Task completed successfully } .addOnFailureListener { // Task failed with an exception }
  52. TEXT RECOGNITION val resultText = result.text for (block in result.textBlocks)

    { val blockText = block.text val blockConfidence = block.confidence val blockLanguages = block.recognizedLanguages val blockCornerPoints = block.cornerPoints val blockFrame = block.boundingBox for (line in block.lines) { val lineText = line.text val lineConfidence = line.confidence val lineLanguages = line.recognizedLanguages val lineCornerPoints = line.cornerPoints val lineFrame = line.boundingBox for (element in line.elements) { val elementText = element.text val elementConfidence = element.confidence val elementLanguages = element.recognizedLanguages val elementCornerPoints = element.cornerPoints val elementFrame = element.boundingBox } } }
  53. AUTO ML

  54. CAMERAX AND AUTO ML dependencies { implementation 'com.google.firebase:firebase-ml-vision:23.0.0' implementation 'com.google.firebase:firebase-ml-vision-automl:18.0.1'

    }
  55. CONFIGURE FIREBASE-HOSTED MODEL val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() val

    remoteModel = FirebaseRemoteModel.Builder("my_remote_model") .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() FirebaseModelManager.getInstance().registerRemoteModel(remoteModel)
  56. DOWNLOAD THE MODEL FirebaseModelManager.getInstance() .downloadRemoteModelIfNeeded(remoteModel) .addOnSuccessListener { // Model downloaded!

    } .addOnFailureListener { // Ooops! }
  57. CONFIGURE THE IMAGE LABLER val labelerOptions = FirebaseVisionOnDeviceAutoMLImageLabelerOptions.Builder() .setRemoteModelName("my_remote_model") .setConfidenceThreshold(threshold)

    .build() val labeler = FirebaseVision.getInstance() .getOnDeviceAutoMLImageLabeler(labelerOptions)
  58. PROCESS IMAGE labeler.processImage(image) .addOnSuccessListener { labels -> for (label in

    labels) { val text = label.text val confidence = label.confidence } }
  59. CONCLUSIONS > CameraX is still in alpha, but ready to

    use today > CameraX Analysis must be synchronous > View finder transform is tricky > ML Kit covers most use cases
  60. THANK YOU FOR LISTENING! SPEAKERDECK.COM/ERIKHELLMAN