Tweet
Tweet

Slide 1

Slide 1 text

PROCESSING CAMERA INPUT ON ANDROID @ERIKHELLMAN SPEAKERDECK.COM/ERIKHELLMAN

Slide 2

Slide 2 text

AGENDA EAN: 0121 7155 Name: Eggs, 6-pack

Slide 3

Slide 3 text

ANDROID CAMERA APIS > android.hardware.Camera (Legacy) > android.hardware.camera2 (Camera 2) > androidx.camera (CameraX)

Slide 4

Slide 4 text

ANDROID CAMERA APIS > android.hardware.Camera (Legacy) > android.hardware.camera2 (Camera 2) > androidx.camera (CameraX)

Slide 5

Slide 5 text

COMPUTER VISION ON ANDROID > ML Kit > ZXing > OpenCV > OpenGL ES > RenderScript > Play Services Vision API

Slide 6

Slide 6 text

DEPRECATED APIS > ML Kit > ZXing > OpenCV > OpenGL ES > RenderScript > Play Services Vision API

Slide 7

Slide 7 text

VIABLE CANDIDATES > ML Kit > OpenCV > OpenGL ES > RenderScript

Slide 8

Slide 8 text

EXCLUDED DUE TO COMPLEXITY > ML Kit > OpenCV > OpenGL ES > RenderScript

Slide 9

Slide 9 text

COMPUTER VISION ON ANDROID: CAMERAX + ML KIT1 1 Also works on iOS!

Slide 10

Slide 10 text

GET STARTED QUICKLY!

Slide 11

Slide 11 text

GRADLE DEPENDENCIES dependencies { // CameraX Core implementation "androidx.camera:camera-core:$camerax_version" // If you want to use Camera2 extensions implementation "androidx.camera:camera-camera2:$camerax_version" // Add Firebase ML Kit for easy computer vision functions! implementation "com.google.firebase:firebase-ml-vision:$firebaseVersion" }

Slide 12

Slide 12 text

SETUP CAMERA PREVIEW val previewConfig = PreviewConfig.Builder() .setLensFacing(CameraX.LensFacing.BACK) .build() val preview = Preview(previewConfig) val viewFinder = findViewById(R.id.viewFinder) preview.setOnPreviewOutputUpdateListener { previewOutput -> viewFinder.surfaceTexture = previewOutput.surfaceTeexture } CameraX.bindToLifecycle(this, preview)

Slide 13

Slide 13 text

SETUP ML KIT FOR BARCODE DETECTION val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options)

Slide 14

Slide 14 text

SETUP CAMERAX IMAGE ANALYZER val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE) .setTargetResolution(Size(1024, 768)) .build() val imageAnalysis = ImageAnalysis(imageAnalysisConfig) imageAnalysis.setAnalyzer { image, rotationDegrees -> // TODO Perform image analysis... } CameraX.bindToLifecycle(this, imageAnalysis)

Slide 15

Slide 15 text

ANALYSE IMAGES WITH ML KIT imageAnalysis.setAnalyzer { image, rotationDegrees -> val imageRotation = degreesToFirebaseRotation(rotationDegrees) image?.image?.let { val visionImage = FirebaseVisionImage.fromMediaImage(it, imageRotation) detector.detectInImage(visionImage) .addOnSuccessListener { barcodes -> displayDetectedBarcode(barcodes) } } }

Slide 16

Slide 16 text

IT WORKS, BUT...

Slide 17

Slide 17 text

VIEW FINDER

Slide 18

Slide 18 text

BAD AND GOOD VIEW FINDER

Slide 19

Slide 19 text

VIEW FINDER LAYOUT

Slide 20

Slide 20 text

VIEW FINDER SKEWED (16:9)

Slide 21

Slide 21 text

VIEW FINDER SKEWED (4:3)

Slide 22

Slide 22 text

VIEW FINDER CORRECT (4:3)

Slide 23

Slide 23 text

WEIRD, BUT ALSO CORRECT

Slide 24

Slide 24 text

GOOGLE CAMERAX SAMPLES (SIMPLIFIED) val matrix = Matrix() val centerX = viewFinderSize.width / 2f val centerY = viewFinderSize.height / 2f val bufferRation = previewSize.height / previewSize.width.toFloat() val scaledHeight = viewFiderSize.width val scaledWidth = Math.roung(viewFinderSize.width * bufferRatio) val xScale = scaledWidth / viewFinderSize.width.toFloat() val yScale = scaledHight / viewFinderSize.height.toFloat() matrix.preScale(xScale, yScale, centerX, centerY)

Slide 25

Slide 25 text

CAMERAX VIEW FINDER

Slide 26

Slide 26 text

SCALE AND CROP! (SIMPLIFIED) val matrix = Matrix() val centerX = viewFinderSize.width / 2f val centerY = viewFinderSize.height / 2f val previewRatio = previewSize.width / previewSize.height.toFloat() val viewFinderRatio = viewFinderSize.width / viewFinderSize.height.toFloat() // Assume view finder is wider than its height matrix.postScale(1.0f, viewFinderRatio * previewRatio, centerX, centerY)

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

TEXTUREVIEW TRANSFORMS ON ANDROID WWW.HELLSOFT.SE/TEXTUREVIEW-TRANSFORMS-ON-ANDROID/

Slide 29

Slide 29 text

YUV COLOR ENCODING

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

−.4 −.3 −.2 −.1 +.1 +.2 +.3 +.4 −.1 +.1 +.2 +.3 +.4 −.2 −.3 −.4 V U

Slide 34

Slide 34 text

YUV ENCODING

Slide 35

Slide 35 text

YUV AND ANDROID CAMERA FRAMES imageAnalysis.setAnalyzer { image, rotationDegrees -> // image.format is always ImageFormat.YUV_420_888 }

Slide 36

Slide 36 text

CONVERT YUV TO RGB (FROM WIKIPEDIA) void YUVImage::yuv2rgb(uint8_t yValue, uint8_t uValue, uint8_t vValue, uint8_t *r, uint8_t *g, uint8_t *b) const { int rTmp = yValue + (1.370705 * (vValue-128)); int gTmp = yValue - (0.698001 * (vValue-128)) - (0.337633 * (uValue-128)); int bTmp = yValue + (1.732446 * (uValue-128)); *r = clamp(rTmp, 0, 255); *g = clamp(gTmp, 0, 255); *b = clamp(bTmp, 0, 255); }

Slide 37

Slide 37 text

GRAYSCALE TO THE RESCUE!

Slide 38

Slide 38 text

EXTRACT GRAYSCALE PIXELS fun extractGrayscaleFromImage(image: Image): Pair { val size = Size(image.width, image.height) val yBuffer = image.planes[0].buffer val grayPixels = ByteBuffer.allocate(yBuffer.capacity()); grayPixels.put(yBuffer) grayPixels.flip() return grayPixels to size }

Slide 39

Slide 39 text

CAMERAX GOTCHAS!

Slide 40

Slide 40 text

CAMERAX ANALYZER MUST BE SYNCHRONOUS!2 imageAnalysis.setAnalyzer { image, rotationDegrees -> // This will crash - analysis must be synchronous! // See https://issuetracker.google.com/issues/139207716 GlobalScope.launch { analyzeImage(image); } } 2 Except when converting to FirebaseVisionImage!

Slide 41

Slide 41 text

CAMERAX ANALYZER CAN'T USE IMAGEWRITER! val imageWriter = ImageWriter(renderScriptSurface, 1) imageAnalysis.setAnalyzer { image, rotationDegrees -> // This will crash - CameraX will call Image.close()! // See https://issuetracker.google.com/issues/139207716 val mediaImage = image?.image if (mediaImage != null) { imageWriter.queueInputImage(image.image) } }

Slide 42

Slide 42 text

ENABLE TORCH (BACK CAMERA) val previewConfig = PreviewConfig.Builder() .setLensFacing(CameraX.LensFacing.BACK) .build() val preview = Preview(previewConfig) preview.enableTorch(true)

Slide 43

Slide 43 text

ANALYZE EVERY IMAGE val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_NEXT_IMAGE) .setImageQueueDepth(5) .setTargetResolution(Size(1024, 768)) .build()

Slide 44

Slide 44 text

ANALYZE IMAGES IN PARALLEL // Thread pool with 5 threads val executor = Executors.newFixedThreadPool(5) val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_NEXT_IMAGE) .setImageQueueDepth(5) .setTargetResolution(Size(1024, 768)) .build() imageAnalysis.setAnalyzer { image, rotationDegrees -> // Analysis can run in parallel } CameraX.bindToLifecycle(this, imageAnalysis)

Slide 45

Slide 45 text

ANALYZE LATEST IMAGE (RECOMMENDED!) val imageAnalysisConfig = ImageAnalysisConfig.Builder() .setImageReaderMode(ImageAnalysis.ImageReaderMode.ACQUIRE_LATEST_IMAGE) .setImageQueueDepth(5) .setTargetResolution(Size(1024, 768)) .build()

Slide 46

Slide 46 text

CAMERAX ANALYSIS API IS TARGET FOR CHANGE!

Slide 47

Slide 47 text

ML KIT

Slide 48

Slide 48 text

FIREBASE ML KIT

Slide 49

Slide 49 text

AUTO-DOWNLOAD REQUIRED ML MODELS (OPTIONAL) ...

Slide 50

Slide 50 text

CONVERT ROTATION DEGREES TO FIREBASE ROTATION fun degreesToFirebaseRotation(degrees: Int): Int = when(degrees) { 0 -> FirebaseVisionImageMetadata.ROTATION_0 90 -> FirebaseVisionImageMetadata.ROTATION_90 180 -> FirebaseVisionImageMetadata.ROTATION_180 270 -> FirebaseVisionImageMetadata.ROTATION_270 else -> throw Exception("Rotation must be 0, 90, 180, or 270.") }

Slide 51

Slide 51 text

TEXT RECOGNITION val detector = FirebaseVision.getInstance().onDeviceTextRecognizer val image = FirebaseVisionImage.fromMediaImage(mediaImage, rotation) val result = detector.processImage(image) .addOnSuccessListener { resultText -> // Task completed successfully } .addOnFailureListener { // Task failed with an exception }

Slide 52

Slide 52 text

TEXT RECOGNITION val resultText = result.text for (block in result.textBlocks) { val blockText = block.text val blockConfidence = block.confidence val blockLanguages = block.recognizedLanguages val blockCornerPoints = block.cornerPoints val blockFrame = block.boundingBox for (line in block.lines) { val lineText = line.text val lineConfidence = line.confidence val lineLanguages = line.recognizedLanguages val lineCornerPoints = line.cornerPoints val lineFrame = line.boundingBox for (element in line.elements) { val elementText = element.text val elementConfidence = element.confidence val elementLanguages = element.recognizedLanguages val elementCornerPoints = element.cornerPoints val elementFrame = element.boundingBox } } }

Slide 53

Slide 53 text

AUTO ML

Slide 54

Slide 54 text

CAMERAX AND AUTO ML dependencies { implementation 'com.google.firebase:firebase-ml-vision:23.0.0' implementation 'com.google.firebase:firebase-ml-vision-automl:18.0.1' }

Slide 55

Slide 55 text

CONFIGURE FIREBASE-HOSTED MODEL val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() val remoteModel = FirebaseRemoteModel.Builder("my_remote_model") .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() FirebaseModelManager.getInstance().registerRemoteModel(remoteModel)

Slide 56

Slide 56 text

DOWNLOAD THE MODEL FirebaseModelManager.getInstance() .downloadRemoteModelIfNeeded(remoteModel) .addOnSuccessListener { // Model downloaded! } .addOnFailureListener { // Ooops! }

Slide 57

Slide 57 text

CONFIGURE THE IMAGE LABLER val labelerOptions = FirebaseVisionOnDeviceAutoMLImageLabelerOptions.Builder() .setRemoteModelName("my_remote_model") .setConfidenceThreshold(threshold) .build() val labeler = FirebaseVision.getInstance() .getOnDeviceAutoMLImageLabeler(labelerOptions)

Slide 58

Slide 58 text

PROCESS IMAGE labeler.processImage(image) .addOnSuccessListener { labels -> for (label in labels) { val text = label.text val confidence = label.confidence } }

Slide 59

Slide 59 text

CONCLUSIONS > CameraX is still in alpha, but ready to use today > CameraX Analysis must be synchronous > View finder transform is tricky > ML Kit covers most use cases

Slide 60

Slide 60 text

THANK YOU FOR LISTENING! SPEAKERDECK.COM/ERIKHELLMAN