How to effectively scan barcodes on Android renaud_mathieu

No content

The history of barcodes

Thesis of Wallace Flint 1932 Norman Woodland and Bernard Silver patented the barcode idea 1949 Philco purchased the barcode patent then sold it to RCA 1952 UPC 1973 First produc scanned at a check-out counter 1974

Philco purchased the barcode patent then sold it to RCA 1952 UPC 1973 First product scanned at a check-out counter 1974 EAN 1975 DataMatrix 1993 QR Code 1994

Symbologies

The mapping between messages and barcodes

More than 100 different barcode symbologies.

Linear Stacked 2D EAN8 EAN13 UPC Code 11 Code 39 Code 93 PDF 417 Code 49 DataMatrix QR Code Aztec

Reading UPC barcodes

No content

No content

No content

Left and Right Patterns for UPC Symbology

No content

No content

No content

No content

No content

No content

CALCULATE A UPC CHECK DIGIT

•Add the digits in the odd-numbered positions together and multiply the total by three 6+2+7+9+1+6=31 31x3=93 •Add the digits in the even-numbered positions 9+7+1+8+1=26 •Add the two results together 93+26=119 •Now what single digit number makes the total a multiple of 10? That’s the check digit. 119 + 1 = 120 Data = 69277198116

No content

Barcode Scanning Use Cases

Scan & Go

Loyalty programs

Lyf Pay

No content

Design guidelines

I/O 2019 - ML Kit x Material Design: Design Patterns for Mobile Machine Learning

Utiliser la caméra pour visualiser le code-barres

Détecter un code-barres

Afficher le résultat

https:"//material.io/collections/machine-learning/

Implement a barcode scanner

Camera Frame Processor Engine

Camera •android.hardware.Camera •android.hardware.Camera2 •CameraX •natario1/CameraView

Frame Processor • CameraX • natario1/CameraView

Engine • ZXing • Firebase ML Kit

Camera Engine CameraView ZXing CameraX MLKit

Camera Engine CameraView ZXing CameraX MLKit Alpha Beta Maintenance ?

No content

implementation 'androidx.lifecycle:lifecycle-runtime-ktx:2.2.0-rc03' implementation 'androidx.camera:camera-core:1.0.0-alpha09' implementation 'androidx.camera:camera-camera2:1.0.0-alpha09' implementation 'androidx.camera:camera-lifecycle:1.0.0-alpha02' implementation 'androidx.camera:camera-view:1.0.0-alpha05' implementation 'com.google.firebase:firebase-analytics:17.2.2' implementation 'com.google.firebase:firebase-ml-vision:24.0.1' implementation 'com.google.firebase:firebase-ml-vision-barcode-model:16.0.2' implementation 'com.otaliastudios:cameraview:2.4.0' implementation 'com.google.zxing:core:3.3.3' implementation 'io.reactivex.rxjava2:rxjava:2.2.9' implementation 'io.reactivex.rxjava2:rxandroid:2.1.1'

@Parcelize data class BarcodeModel( val value: String, val symbology: Symbology ) : Parcelable Models

enum class Symbology(val value: String) { AZTEC("aztec"), CODE128("code128"), CODE39("code39"), CODE93("code93"), CODABAR("codabar"), DATAMATRIX("datamatrix"), EAN13("ean13"), EAN8("ean8"), ITF14("itf14"), QR("qr"), UPCA("upca"), UPCE("upce"), PDF417("pdf417"); } Models

CameraView + ZXing

class OCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null ) : CameraView(context, attrs) { private var isPaused: Boolean = false private val frameProcessor = FrameProcessor { if (!isPaused) { "// I’ve got a new frame } } init { setLifecycleOwner(context as LifecycleOwner) }

} init { setLifecycleOwner(context as LifecycleOwner) } @OnLifecycleEvent(Lifecycle.Event.ON_RESUME) private fun resumeScan() { isPaused = false addFrameProcessor(frameProcessor) } @OnLifecycleEvent(Lifecycle.Event.ON_PAUSE) private fun pauseScan() { isPaused = true removeFrameProcessor(frameProcessor) }

interface BarcodeAnalyzer { fun onBarcodeScanned(): Observable }

!!/** * MultiFormatReader is a convenience class and the main entry point into the library for most uses. * By default it attempts to decode all barcode formats that the library supports. Optionally, you * can provide a hints object to request different behavior, for example only decoding QR codes. * * @author Sean Owen * @author dswitkin@google.com (Daniel Switkin) !*/ public final class MultiFormatReader implements Reader { … }

!!/** * Decode an image using the state set up by calling setHints() previously. Continuous scan * clients will get a large! speed increase by using this instead of decode(). * * @param image The pixel data to decode * @return The contents of the image * @throws NotFoundException Any errors which occurred !*/ public Result decodeWithState(BinaryBitmap image) throws NotFoundException { "// Make sure to set up the default state so we don't crash if (readers "== null) { setHints(null); } return decodeInternal(image); }

!!/** * This class is the core bitmap class used by ZXing to represent 1 bit data. Reader objects * accept a BinaryBitmap and attempt to decode it. * * @author dswitkin@google.com (Daniel Switkin) !*/ public final class BinaryBitmap { private final Binarizer binarizer; private BitMatrix matrix; public BinaryBitmap(Binarizer binarizer) { if (binarizer "== null) { throw new IllegalArgumentException("Binarizer must be non-null."); } this.binarizer = binarizer; }

!!/** * This class implements a local thresholding algorithm, which while slower than the * GlobalHistogramBinarizer, is fairly efficient for what it does. It is designed for * high frequency images of barcodes with black data on white backgrounds. For this application, * it does a much better job than a global blackpoint with severe shadows and gradients. * However it tends to produce artifacts on lower frequency images and is therefore not * a good general purpose binarizer for uses outside ZXing. * * This class extends GlobalHistogramBinarizer, using the older histogram approach for 1D readers, * and the newer local approach for 2D readers. 1D decoding using a per-row histogram is already * inherently local, and only fails for horizontal gradients. We can revisit that problem later, * but for now it was not a win to use local blocks for 1D. * * This Binarizer is the default for the unit tests and the recommended class for library users. * * @author dswitkin@google.com (Daniel Switkin) !*/ public final class HybridBinarizer extends GlobalHistogramBinarizer { public HybridBinarizer(LuminanceSource source) { super(source); } }

!!/** * The purpose of this class hierarchy is to abstract different bitmap implementations across * platforms into a standard interface for requesting greyscale luminance values. The interface * only provides immutable methods; therefore crop and rotation create copies. This is to ensure * that one Reader does not modify the original luminance source and leave it in an unknown state * for other Readers in the chain. * * @author dswitkin@google.com (Daniel Switkin) !*/ public abstract class LuminanceSource { private final int width; private final int height; protected LuminanceSource(int width, int height) { this.width = width; this.height = height; }

!!/** * This object extends LuminanceSource around an array of YUV data returned from the camera driver, * with the option to crop to a rectangle within the full data. This can be used to exclude * superfluous pixels around the perimeter and speed up decoding. * * It works for any pixel format where the Y channel is planar and appears first, including * YCbCr_420_SP and YCbCr_422_SP. * * @author dswitkin@google.com (Daniel Switkin) !*/ public final class PlanarYUVLuminanceSource extends LuminanceSource {

private fun detectInImageSingleOrientation( data: ByteArray, width: Int, height: Int, rotation: Int ): Result? { val source = getPlanarYUVLuminanceSource(data, width, height, rotation) val bitmap = BinaryBitmap(HybridBinarizer(source)) var result: Result? = null try { result = multiFormatReader.decodeWithState(bitmap) } catch (re: NotFoundException) { } finally { multiFormatReader.reset() } return result }

private fun detectInImage(data: ByteArray, width: Int, height: Int): Observable = Observable.create { emitter "-> val result = detectInImageSingleOrientation(data, width, height, ROTATION_90) "?: detectInImageSingleOrientation(data, width, height, ROTATION_0) if (result "!= null) { emitter.onNext(result) } }

class ZXingImageAnalyzer : BarcodeAnalyzer { companion object { const val ROTATION_0 = 0 const val ROTATION_90 = 90 } private val multiFormatReader: MultiFormatReader = MultiFormatReader() val onFrameProcessorPublisher: PublishSubject = PublishSubject.create() override fun onBarcodeScanned(): Observable = onFrameProcessorPublisher .flatMap { detectInImage(it.data, it.size.width, it.size.height) } .map { BarcodeModel(it.text, Symbology.CODE128) }

class OCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null ) : CameraView(context, attrs), BarcodeProcessor { private var isPaused: Boolean = false private val zxingImageAnalyzer = ZXingImageAnalyzer() private val frameProcessor = FrameProcessor { if (!isPaused) { zxingImageAnalyzer.onFrameProcessorPublisher.onNext(it) } }

CameraX ♥ MLKit

The CameraX library is in alpha stage, as its API surfaces aren't yet finalized.

CameraX is an addition to Jetpack that makes it easier to leverage the capabilities of Camera2 APIs.

Use cases : • Implement preview • Analyze images • Take a photo

Use cases can be combined and active concurrently.

class ScanApplication : Application(), CameraXConfig.Provider { override fun getCameraXConfig(): CameraXConfig = Camera2Config.defaultConfig() }

!!/** * Convenience class for generating a pre-populated Camera2 {@link CameraXConfig}. !*/ public final class Camera2Config { private Camera2Config() { } !!/** * Creates a {@link CameraXConfig} containing the default Camera2 implementation for CameraX. !*/ @NonNull public static CameraXConfig defaultConfig() { "// Create the camera factory for creating Camera2 camera objects CameraFactory.Provider cameraFactoryProvider = Camera2CameraFactory"::new; "// Create the DeviceSurfaceManager for Camera2 CameraDeviceSurfaceManager.Provider surfaceManagerProvider = Camera2DeviceSurfaceManager"::new; "// Create default configuration factory UseCaseConfigFactory.Provider configFactoryProvider = context "-> { ExtendableUseCaseConfigFactory factory = new ExtendableUseCaseConfigFactory(); factory.installDefaultProvider( ImageAnalysisConfig.class, new ImageAnalysisConfigProvider(context));

CameraDeviceSurfaceManager.Provider surfaceManagerProvider = Camera2DeviceSurfaceManager"::new; "// Create default configuration factory UseCaseConfigFactory.Provider configFactoryProvider = context "-> { ExtendableUseCaseConfigFactory factory = new ExtendableUseCaseConfigFactory(); factory.installDefaultProvider( ImageAnalysisConfig.class, new ImageAnalysisConfigProvider(context)); factory.installDefaultProvider( ImageCaptureConfig.class, new ImageCaptureConfigProvider(context)); factory.installDefaultProvider( VideoCaptureConfig.class, new VideoCaptureConfigProvider(context)); factory.installDefaultProvider( PreviewConfig.class, new PreviewConfigProvider(context)); return factory; }; CameraXConfig.Builder appConfigBuilder = new CameraXConfig.Builder() .setCameraFactoryProvider(cameraFactoryProvider) .setDeviceSurfaceManagerProvider(surfaceManagerProvider) .setUseCaseConfigFactoryProvider(configFactoryProvider); return appConfigBuilder.build(); } }

class XCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null ) : PreviewView(context, attrs), BarcodeProcessor { private var cameraProviderFuture: ListenableFuture = ProcessCameraProvider.getInstance(context) private val cameraSelector: CameraSelector = CameraSelector.Builder() .requireLensFacing(CameraSelector.LENS_FACING_BACK) .build() init { cameraProviderFuture.addListener( Runnable { bindToLifecycle(cameraProviderFuture.get()) }, ContextCompat.getMainExecutor(context) ) }

public Camera bindToLifecycle(@NonNull LifecycleOwner lifecycleOwner, @NonNull CameraSelector cameraSelector, @NonNull UseCase""... useCases) { return CameraX.bindToLifecycle(lifecycleOwner, cameraSelector, useCases); }

private fun bindToLifecycle(cameraProvider: ProcessCameraProvider) { val preview: Preview = Preview.Builder() .setTargetName("Preview") .build() preview.previewSurfaceProvider = previewSurfaceProvider cameraProvider.bindToLifecycle( context as LifecycleOwner, cameraSelector, preview ) }

ML Kit's barcode scanning API •Reads most standard formats •Automatic format detection •Extracts structured data •Works with any orientation •Runs on the device

Version 24.0.0 of firebase-ml- vision introduces a new barcode scanning model

ML Kit is still in Beta ML Kit still requires a Firebase project

class FirebaseImageAnalyzer : BarcodeAnalyzer { private val onBarcodeScannedPublisher: PublishSubject = PublishSubject.create() override fun onBarcodeScanned(): Observable = onBarcodeScannedPublisher

class FirebaseImageAnalyzer : BarcodeAnalyzer { private val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() private val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) private val onBarcodeScannedPublisher: PublishSubject = PublishSubject.create() override fun onBarcodeScanned(): Observable = onBarcodeScannedPublisher

class FirebaseImageAnalyzer : ImageAnalysis.Analyzer, BarcodeAnalyzer { private val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() private val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) private val onBarcodeScannedPublisher: PublishSubject = PublishSubject.create() override fun analyze(image: ImageProxy) { } override fun onBarcodeScanned(): Observable = onBarcodeScannedPublisher private fun degreesToFirebaseRotation(degrees: Int): Int = when (degrees) { 0 "-> FirebaseVisionImageMetadata.ROTATION_0 90 "-> FirebaseVisionImageMetadata.ROTATION_90 180 "-> FirebaseVisionImageMetadata.ROTATION_180 270 "-> FirebaseVisionImageMetadata.ROTATION_270 else "-> throw Exception("Rotation must be 0, 90, 180, or 270.") }

private fun bindToLifecycle(cameraProvider: ProcessCameraProvider) { val preview: Preview = Preview.Builder() .setTargetName("Preview") .build() preview.previewSurfaceProvider = previewSurfaceProvider val imageAnalysis = ImageAnalysis.Builder() .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setTargetResolution((Size(1280, 720))) .build() imageAnalysis.setAnalyzer( Executors.newSingleThreadExecutor(), firebaseImageAnalyzer ) cameraProvider.bindToLifecycle( context as LifecycleOwner, cameraSelector, preview, imageAnalysis ) }

override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }

override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }

override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }

override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }

!!/** * Only deliver the latest image to the analyzer, dropping images as they arrive. * *

This strategy ignores the value set by {@link Builder#setImageQueueDepth(int)}. * Only one image will be delivered for analysis at a time. If more images are produced * while that image is being analyzed, they will be dropped and not queued for delivery. * Once the image being analyzed is closed by calling {@link ImageProxy#close()}, the * next latest image will be delivered. * *

Internally this strategy may make use of an internal {@link Executor} to receive * and drop images from the producer. A performance-tuned executor will be created * internally unless one is explicitly provided by * {@link Builder#setBackgroundExecutor(Executor)}. In order to * ensure smooth operation of this backpressure strategy, any user supplied * {@link Executor} must be able to quickly respond to tasks posted to it, so setting * the executor manually should only be considered in advanced use cases. * * @see Builder#setBackgroundExecutor(Executor) !*/ public static final int STRATEGY_KEEP_ONLY_LATEST = 0;

class ScanActivity : AppCompatActivity() { companion object { const val VIBRATION_DURATION: Long = 250 } private val compositeDisposable = CompositeDisposable() override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) compositeDisposable.add(previewView.onBarcodeScanned() .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .subscribe( { vibrate() }, { Log.e("ScanActivity", "Scanning error", it) } ) ) }

"

"

Go beyond…

Android Dev Summit ’19 https:"//www.youtube.com/watch?v=CSJMxEOgiJQ

• Barcode is far away • Barcode is distorted source: Android Dev Summit ’19

source: Android Dev Summit ’19

source: Android Dev Summit ’19

source: Android Dev Summit ’19

source: Android Dev Summit ’19

Machine Learning?

No content

No content

Reading barcodes with neural networks Department of Electrical Engineering Linköping University SE-581 83 Linköping, Sweden Copyright © 2017 Fredrik Fridborn

No content

No content

No content

No content

No content

No content

No content

No content

Thank you