$30 off During Our Annual Pro Sale. View Details »

Processing camera input on Android

Processing camera input on Android

Processing and analysing images from the camera in an Android application has always been a complicated thing to solve. Despite Google making a big overhaul of the camera APIs with camera2, developers still struggled with even the simplest tasks such as scanning QR-codes or doing text detection.

At Google IO 2019, we got a new library from Google to make our work easier. The new library is called CameraX and significantly simplifies the common tasks we have when working with the camera on Android. In this session, we will learn how to use the CameraX API for any kind of processing and computer vision. After this session, you’ll feel much more confident to implement the camera processing the way you wanted, without having to resort to heavy libraries that rely on the legacy APIs.

Erik Hellman

October 03, 2019
Tweet

More Decks by Erik Hellman

Other Decks in Programming

Transcript

  1. PROCESSING
    CAMERA INPUT
    ON ANDROID
    @ERIKHELLMAN
    SPEAKERDECK.COM/ERIKHELLMAN

    View Slide

  2. AGENDA
    EAN: 0121 7155
    Name: Eggs, 6-pack

    View Slide

  3. ANDROID CAMERA APIS
    > android.hardware.Camera (Legacy)
    > android.hardware.camera2 (Camera 2)
    > androidx.camera (CameraX)

    View Slide

  4. ANDROID CAMERA APIS
    > android.hardware.Camera (Legacy)
    > android.hardware.camera2 (Camera 2)
    > androidx.camera (CameraX)

    View Slide

  5. COMPUTER VISION ON ANDROID
    > ML Kit
    > ZXing
    > OpenCV
    > OpenGL ES
    > RenderScript
    > Play Services Vision API

    View Slide

  6. DEPRECATED APIS
    > ML Kit
    > ZXing
    > OpenCV
    > OpenGL ES
    > RenderScript
    > Play Services Vision API

    View Slide

  7. VIABLE CANDIDATES
    > ML Kit
    > OpenCV
    > OpenGL ES
    > RenderScript

    View Slide

  8. EXCLUDED DUE TO COMPLEXITY
    > ML Kit
    > OpenCV
    > OpenGL ES
    > RenderScript

    View Slide

  9. COMPUTER VISION ON ANDROID:
    CAMERAX + ML KIT1
    1 Also works on iOS!

    View Slide

  10. GET STARTED QUICKLY!

    View Slide

  11. GRADLE DEPENDENCIES
    dependencies {
    // CameraX Core
    implementation "androidx.camera:camera-core:$camerax_version"
    // If you want to use Camera2 extensions
    implementation "androidx.camera:camera-camera2:$camerax_version"
    // Add Firebase ML Kit for easy computer vision functions!
    implementation "com.google.firebase:firebase-ml-vision:$firebaseVersion"
    }

    View Slide

  12. SETUP CAMERA PREVIEW
    val previewConfig = PreviewConfig.Builder()
    .setLensFacing(CameraX.LensFacing.BACK)
    .build()
    val preview = Preview(previewConfig)
    val viewFinder = findViewById(R.id.viewFinder)
    preview.setOnPreviewOutputUpdateListener { previewOutput ->
    viewFinder.surfaceTexture = previewOutput.surfaceTeexture
    }
    CameraX.bindToLifecycle(this, preview)

    View Slide

  13. SETUP ML KIT FOR BARCODE DETECTION
    val options = FirebaseVisionBarcodeDetectorOptions.Builder()
    .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS)
    .build()
    val detector = FirebaseVision.getInstance()
    .getVisionBarcodeDetector(options)

    View Slide

  14. SETUP CAMERAX IMAGE ANALYZER
    val imageAnalysisConfig = ImageAnalysisConfig.Builder()
    .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
    .setTargetResolution(Size(1024, 768))
    .build()
    val imageAnalysis = ImageAnalysis(imageAnalysisConfig)
    imageAnalysis.setAnalyzer { image, rotationDegrees ->
    // TODO Perform image analysis...
    }
    CameraX.bindToLifecycle(this, imageAnalysis)

    View Slide

  15. ANALYSE IMAGES WITH ML KIT
    imageAnalysis.setAnalyzer { image, rotationDegrees ->
    val imageRotation = degreesToFirebaseRotation(rotationDegrees)
    image?.image?.let {
    val visionImage = FirebaseVisionImage.fromMediaImage(it, imageRotation)
    detector.detectInImage(visionImage)
    .addOnSuccessListener { barcodes ->
    displayDetectedBarcode(barcodes)
    }
    }
    }

    View Slide

  16. IT WORKS, BUT...

    View Slide

  17. VIEW FINDER

    View Slide

  18. BAD AND GOOD VIEW FINDER

    View Slide

  19. VIEW FINDER LAYOUT
    android:id="@+id/viewFinder"
    android:layout_width="0dp"
    android:layout_height="0dp"
    app:layout_constraintDimensionRatio="16:9"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintHorizontal_bias="0.0"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintTop_toTopOf="parent" />

    View Slide

  20. VIEW FINDER SKEWED (16:9)

    View Slide

  21. VIEW FINDER SKEWED (4:3)

    View Slide

  22. VIEW FINDER CORRECT (4:3)

    View Slide

  23. WEIRD, BUT ALSO CORRECT

    View Slide

  24. GOOGLE CAMERAX SAMPLES (SIMPLIFIED)
    val matrix = Matrix()
    val centerX = viewFinderSize.width / 2f
    val centerY = viewFinderSize.height / 2f
    val bufferRation = previewSize.height / previewSize.width.toFloat()
    val scaledHeight = viewFiderSize.width
    val scaledWidth = Math.roung(viewFinderSize.width * bufferRatio)
    val xScale = scaledWidth / viewFinderSize.width.toFloat()
    val yScale = scaledHight / viewFinderSize.height.toFloat()
    matrix.preScale(xScale, yScale, centerX, centerY)

    View Slide

  25. CAMERAX VIEW FINDER

    View Slide

  26. SCALE AND CROP! (SIMPLIFIED)
    val matrix = Matrix()
    val centerX = viewFinderSize.width / 2f
    val centerY = viewFinderSize.height / 2f
    val previewRatio = previewSize.width / previewSize.height.toFloat()
    val viewFinderRatio = viewFinderSize.width / viewFinderSize.height.toFloat()
    // Assume view finder is wider than its height
    matrix.postScale(1.0f, viewFinderRatio * previewRatio, centerX, centerY)

    View Slide

  27. View Slide

  28. TEXTUREVIEW TRANSFORMS
    ON ANDROID
    WWW.HELLSOFT.SE/TEXTUREVIEW-TRANSFORMS-ON-ANDROID/

    View Slide

  29. YUV COLOR
    ENCODING

    View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. −.4 −.3 −.2 −.1 +.1 +.2 +.3 +.4
    −.1
    +.1
    +.2
    +.3
    +.4
    −.2
    −.3
    −.4
    V
    U

    View Slide

  34. YUV ENCODING

    View Slide

  35. YUV AND ANDROID CAMERA FRAMES
    imageAnalysis.setAnalyzer { image, rotationDegrees ->
    // image.format is always ImageFormat.YUV_420_888
    }

    View Slide

  36. CONVERT YUV TO RGB (FROM WIKIPEDIA)
    void YUVImage::yuv2rgb(uint8_t yValue, uint8_t uValue, uint8_t vValue,
    uint8_t *r, uint8_t *g, uint8_t *b) const {
    int rTmp = yValue + (1.370705 * (vValue-128));
    int gTmp = yValue - (0.698001 * (vValue-128)) - (0.337633 * (uValue-128));
    int bTmp = yValue + (1.732446 * (uValue-128));
    *r = clamp(rTmp, 0, 255);
    *g = clamp(gTmp, 0, 255);
    *b = clamp(bTmp, 0, 255);
    }

    View Slide

  37. GRAYSCALE TO THE RESCUE!

    View Slide

  38. EXTRACT GRAYSCALE PIXELS
    fun extractGrayscaleFromImage(image: Image): Pair {
    val size = Size(image.width, image.height)
    val yBuffer = image.planes[0].buffer
    val grayPixels = ByteBuffer.allocate(yBuffer.capacity());
    grayPixels.put(yBuffer)
    grayPixels.flip()
    return grayPixels to size
    }

    View Slide

  39. CAMERAX
    GOTCHAS!

    View Slide

  40. CAMERAX ANALYZER
    MUST BE SYNCHRONOUS!2
    imageAnalysis.setAnalyzer { image, rotationDegrees ->
    // This will crash - analysis must be synchronous!
    // See https://issuetracker.google.com/issues/139207716
    GlobalScope.launch { analyzeImage(image); }
    }
    2 Except when converting to FirebaseVisionImage!

    View Slide

  41. CAMERAX ANALYZER
    CAN'T USE IMAGEWRITER!
    val imageWriter = ImageWriter(renderScriptSurface, 1)
    imageAnalysis.setAnalyzer { image, rotationDegrees ->
    // This will crash - CameraX will call Image.close()!
    // See https://issuetracker.google.com/issues/139207716
    val mediaImage = image?.image
    if (mediaImage != null) {
    imageWriter.queueInputImage(image.image)
    }
    }

    View Slide

  42. ENABLE TORCH (BACK CAMERA)
    val previewConfig = PreviewConfig.Builder()
    .setLensFacing(CameraX.LensFacing.BACK)
    .build()
    val preview = Preview(previewConfig)
    preview.enableTorch(true)

    View Slide

  43. ANALYZE EVERY IMAGE
    val imageAnalysisConfig = ImageAnalysisConfig.Builder()
    .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_NEXT_IMAGE)
    .setImageQueueDepth(5)
    .setTargetResolution(Size(1024, 768))
    .build()

    View Slide

  44. ANALYZE IMAGES IN PARALLEL
    // Thread pool with 5 threads
    val executor = Executors.newFixedThreadPool(5)
    val imageAnalysisConfig = ImageAnalysisConfig.Builder()
    .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_NEXT_IMAGE)
    .setImageQueueDepth(5)
    .setTargetResolution(Size(1024, 768))
    .build()
    imageAnalysis.setAnalyzer { image, rotationDegrees ->
    // Analysis can run in parallel
    }
    CameraX.bindToLifecycle(this, imageAnalysis)

    View Slide

  45. ANALYZE LATEST IMAGE (RECOMMENDED!)
    val imageAnalysisConfig = ImageAnalysisConfig.Builder()
    .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE)
    .setImageQueueDepth(5)
    .setTargetResolution(Size(1024, 768))
    .build()

    View Slide

  46. CAMERAX ANALYSIS API
    IS TARGET FOR CHANGE!

    View Slide

  47. ML KIT

    View Slide

  48. FIREBASE ML KIT

    View Slide

  49. AUTO-DOWNLOAD REQUIRED ML MODELS
    (OPTIONAL)

    ...
    android:name="com.google.firebase.ml.vision.DEPENDENCIES"
    android:value="ocr" />


    View Slide

  50. CONVERT ROTATION DEGREES
    TO FIREBASE ROTATION
    fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) {
    0 -> FirebaseVisionImageMetadata.ROTATION_0
    90 -> FirebaseVisionImageMetadata.ROTATION_90
    180 -> FirebaseVisionImageMetadata.ROTATION_180
    270 -> FirebaseVisionImageMetadata.ROTATION_270
    else -> throw Exception("Rotation must be 0, 90, 180, or 270.")
    }

    View Slide

  51. TEXT RECOGNITION
    val detector = FirebaseVision.getInstance().onDeviceTextRecognizer
    val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation)
    val result = detector.processImage(image)
    .addOnSuccessListener { resultText ->
    // Task completed successfully
    }
    .addOnFailureListener {
    // Task failed with an exception
    }

    View Slide

  52. TEXT RECOGNITION
    val resultText = result.text
    for (block in result.textBlocks) {
    val blockText = block.text
    val blockConfidence = block.confidence
    val blockLanguages = block.recognizedLanguages
    val blockCornerPoints = block.cornerPoints
    val blockFrame = block.boundingBox
    for (line in block.lines) {
    val lineText = line.text
    val lineConfidence = line.confidence
    val lineLanguages = line.recognizedLanguages
    val lineCornerPoints = line.cornerPoints
    val lineFrame = line.boundingBox
    for (element in line.elements) {
    val elementText = element.text
    val elementConfidence = element.confidence
    val elementLanguages = element.recognizedLanguages
    val elementCornerPoints = element.cornerPoints
    val elementFrame = element.boundingBox
    }
    }
    }

    View Slide

  53. AUTO ML

    View Slide

  54. CAMERAX AND AUTO ML
    dependencies {
    implementation 'com.google.firebase:firebase-ml-vision:23.0.0'
    implementation 'com.google.firebase:firebase-ml-vision-automl:18.0.1'
    }

    View Slide

  55. CONFIGURE FIREBASE-HOSTED MODEL
    val conditions = FirebaseModelDownloadConditions.Builder()
    .requireWifi()
    .build()
    val remoteModel = FirebaseRemoteModel.Builder("my_remote_model")
    .enableModelUpdates(true)
    .setInitialDownloadConditions(conditions)
    .setUpdatesDownloadConditions(conditions)
    .build()
    FirebaseModelManager.getInstance().registerRemoteModel(remoteModel)

    View Slide

  56. DOWNLOAD THE MODEL
    FirebaseModelManager.getInstance()
    .downloadRemoteModelIfNeeded(remoteModel)
    .addOnSuccessListener {
    // Model downloaded!
    }
    .addOnFailureListener {
    // Ooops!
    }

    View Slide

  57. CONFIGURE THE IMAGE LABLER
    val labelerOptions = FirebaseVisionOnDeviceAutoMLImageLabelerOptions.Builder()
    .setRemoteModelName("my_remote_model")
    .setConfidenceThreshold(threshold)
    .build()
    val labeler = FirebaseVision.getInstance()
    .getOnDeviceAutoMLImageLabeler(labelerOptions)

    View Slide

  58. PROCESS IMAGE
    labeler.processImage(image)
    .addOnSuccessListener { labels ->
    for (label in labels) {
    val text = label.text
    val confidence = label.confidence
    }
    }

    View Slide

  59. CONCLUSIONS
    > CameraX is still in alpha, but ready to use today
    > CameraX Analysis must be synchronous
    > View finder transform is tricky
    > ML Kit covers most use cases

    View Slide

  60. THANK YOU FOR
    LISTENING!
    SPEAKERDECK.COM/ERIKHELLMAN

    View Slide