How to effectively scan barcodes on Android

Android does not provide a native solution to detect barcodes in your application. However, some barcode image processing libraries are available but it is not always easy to use it for top performance.

Through this talk, we will see how to implement barcode detection in your Android application from camera management to optimization of barcode recognition thanks to machine learning.

Transcript

1. How to effectively scan barcodes on Android renaud_mathieu
3. The history of barcodes
4. Thesis of Wallace Flint 1932 Norman Woodland and Bernard Silver patented the barcode idea 1949 Philco purchased the barcode patent then sold it to RCA 1952 UPC 1973 First produc scanned at a check-out counter 1974
5. Philco purchased the barcode patent then sold it to RCA 1952 UPC 1973 First product scanned at a check-out counter 1974 EAN 1975 DataMatrix 1993 QR Code 1994
6. Symbologies
7. The mapping between messages and barcodes
8. More than 100 different barcode symbologies.
9. Linear Stacked 2D EAN8 EAN13 UPC Code 11 Code 39 Code 93 PDF 417 Code 49 DataMatrix QR Code Aztec
10. Reading UPC barcodes
14. Left and Right Patterns for UPC Symbology
21. CALCULATE A UPC CHECK DIGIT
22. •Add the digits in the odd-numbered positions together and multiply the total by three 6+2+7+9+1+6=31 31x3=93 •Add the digits in the even-numbered positions 9+7+1+8+1=26 •Add the two results together 93+26=119 •Now what single digit number makes the total a multiple of 10? That’s the check digit. 119 + 1 = 120 Data = 69277198116
24. Barcode Scanning Use Cases
25. Scan & Go
26. Loyalty programs
27. Lyf Pay
29. Design guidelines
30. I/O 2019 - ML Kit x Material Design: Design Patterns for Mobile Machine Learning
31. Utiliser la caméra pour visualiser le code-barres
32. Détecter un code-barres
33. Afficher le résultat
34. https:"//material.io/collections/machine-learning/
35. Implement a barcode scanner
36. Camera Frame Processor Engine
37. Camera •android.hardware.Camera •android.hardware.Camera2 •CameraX •natario1/CameraView
38. Frame Processor • CameraX • natario1/CameraView
39. Engine • ZXing • Firebase ML Kit
40. Camera Engine CameraView ZXing CameraX MLKit
41. Camera Engine CameraView ZXing CameraX MLKit Alpha Beta Maintenance ?
43. implementation 'androidx.lifecycle:lifecycle-runtime-ktx:2.2.0-rc03' implementation 'androidx.camera:camera-core:1.0.0-alpha09' implementation 'androidx.camera:camera-camera2:1.0.0-alpha09' implementation 'androidx.camera:camera-lifecycle:1.0.0-alpha02' implementation 'androidx.camera:camera-view:1.0.0-alpha05' implementation 'com.google.firebase:firebase-analytics:17.2.2' implementation 'com.google.firebase:firebase-ml-vision:24.0.1' implementation 'com.google.firebase:firebase-ml-vision-barcode-model:16.0.2' implementation 'com.otaliastudios:cameraview:2.4.0' implementation 'com.google.zxing:core:3.3.3' implementation 'io.reactivex.rxjava2:rxjava:2.2.9' implementation 'io.reactivex.rxjava2:rxandroid:2.1.1'
44. @Parcelize data class BarcodeModel( val value: String, val symbology: Symbology ) : Parcelable Models
45. enum class Symbology(val value: String) { AZTEC("aztec"), CODE128("code128"), CODE39("code39"), CODE93("code93"), CODABAR("codabar"), DATAMATRIX("datamatrix"), EAN13("ean13"), EAN8("ean8"), ITF14("itf14"), QR("qr"), UPCA("upca"), UPCE("upce"), PDF417("pdf417"); } Models
46. CameraView + ZXing
47. class OCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null ) : CameraView(context, attrs) { private var isPaused: Boolean = false private val frameProcessor = FrameProcessor { if (!isPaused) { "// I’ve got a new frame } } init { setLifecycleOwner(context as LifecycleOwner) }
48. } init { setLifecycleOwner(context as LifecycleOwner) } @OnLifecycleEvent(Lifecycle.Event.ON_RESUME) private fun resumeScan() { isPaused = false addFrameProcessor(frameProcessor) } @OnLifecycleEvent(Lifecycle.Event.ON_PAUSE) private fun pauseScan() { isPaused = true removeFrameProcessor(frameProcessor) }
49. interface BarcodeAnalyzer { fun onBarcodeScanned(): Observable<BarcodeModel> }
50. !!/** * MultiFormatReader is a convenience class and the main entry point into the library for most uses. * By default it attempts to decode all barcode formats that the library supports. Optionally, you * can provide a hints object to request different behavior, for example only decoding QR codes. * * @author Sean Owen * @author dswitkin@google.com (Daniel Switkin) !*/ public final class MultiFormatReader implements Reader { … }
51. !!/** * Decode an image using the state set up by calling setHints() previously. Continuous scan * clients will get a <b>large!</b> speed increase by using this instead of decode(). * * @param image The pixel data to decode * @return The contents of the image * @throws NotFoundException Any errors which occurred !*/ public Result decodeWithState(BinaryBitmap image) throws NotFoundException { "// Make sure to set up the default state so we don't crash if (readers "== null) { setHints(null); } return decodeInternal(image); }
52. !!/** * This class is the core bitmap class used by ZXing to represent 1 bit data. Reader objects * accept a BinaryBitmap and attempt to decode it. * * @author dswitkin@google.com (Daniel Switkin) !*/ public final class BinaryBitmap { private final Binarizer binarizer; private BitMatrix matrix; public BinaryBitmap(Binarizer binarizer) { if (binarizer "== null) { throw new IllegalArgumentException("Binarizer must be non-null."); } this.binarizer = binarizer; }
53. !!/** * This class implements a local thresholding algorithm, which while slower than the * GlobalHistogramBinarizer, is fairly efficient for what it does. It is designed for * high frequency images of barcodes with black data on white backgrounds. For this application, * it does a much better job than a global blackpoint with severe shadows and gradients. * However it tends to produce artifacts on lower frequency images and is therefore not * a good general purpose binarizer for uses outside ZXing. * * This class extends GlobalHistogramBinarizer, using the older histogram approach for 1D readers, * and the newer local approach for 2D readers. 1D decoding using a per-row histogram is already * inherently local, and only fails for horizontal gradients. We can revisit that problem later, * but for now it was not a win to use local blocks for 1D. * * This Binarizer is the default for the unit tests and the recommended class for library users. * * @author dswitkin@google.com (Daniel Switkin) !*/ public final class HybridBinarizer extends GlobalHistogramBinarizer { public HybridBinarizer(LuminanceSource source) { super(source); } }
54. !!/** * The purpose of this class hierarchy is to abstract different bitmap implementations across * platforms into a standard interface for requesting greyscale luminance values. The interface * only provides immutable methods; therefore crop and rotation create copies. This is to ensure * that one Reader does not modify the original luminance source and leave it in an unknown state * for other Readers in the chain. * * @author dswitkin@google.com (Daniel Switkin) !*/ public abstract class LuminanceSource { private final int width; private final int height; protected LuminanceSource(int width, int height) { this.width = width; this.height = height; }
55. !!/** * This object extends LuminanceSource around an array of YUV data returned from the camera driver, * with the option to crop to a rectangle within the full data. This can be used to exclude * superfluous pixels around the perimeter and speed up decoding. * * It works for any pixel format where the Y channel is planar and appears first, including * YCbCr_420_SP and YCbCr_422_SP. * * @author dswitkin@google.com (Daniel Switkin) !*/ public final class PlanarYUVLuminanceSource extends LuminanceSource {
56. private fun detectInImageSingleOrientation( data: ByteArray, width: Int, height: Int, rotation: Int ): Result? { val source = getPlanarYUVLuminanceSource(data, width, height, rotation) val bitmap = BinaryBitmap(HybridBinarizer(source)) var result: Result? = null try { result = multiFormatReader.decodeWithState(bitmap) } catch (re: NotFoundException) { } finally { multiFormatReader.reset() } return result }
57. private fun detectInImage(data: ByteArray, width: Int, height: Int): Observable<Result> = Observable.create { emitter "-> val result = detectInImageSingleOrientation(data, width, height, ROTATION_90) "?: detectInImageSingleOrientation(data, width, height, ROTATION_0) if (result "!= null) { emitter.onNext(result) } }
58. class ZXingImageAnalyzer : BarcodeAnalyzer { companion object { const val ROTATION_0 = 0 const val ROTATION_90 = 90 } private val multiFormatReader: MultiFormatReader = MultiFormatReader() val onFrameProcessorPublisher: PublishSubject<Frame> = PublishSubject.create() override fun onBarcodeScanned(): Observable<BarcodeModel> = onFrameProcessorPublisher .flatMap { detectInImage(it.data, it.size.width, it.size.height) } .map { BarcodeModel(it.text, Symbology.CODE128) }
59. class OCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null ) : CameraView(context, attrs), BarcodeProcessor { private var isPaused: Boolean = false private val zxingImageAnalyzer = ZXingImageAnalyzer() private val frameProcessor = FrameProcessor { if (!isPaused) { zxingImageAnalyzer.onFrameProcessorPublisher.onNext(it) } }
60. CameraX ♥ MLKit
61. The CameraX library is in alpha stage, as its API surfaces aren't yet finalized.
62. CameraX is an addition to Jetpack that makes it easier to leverage the capabilities of Camera2 APIs.
63. Use cases : • Implement preview • Analyze images • Take a photo
64. Use cases can be combined and active concurrently.
65. class ScanApplication : Application(), CameraXConfig.Provider { override fun getCameraXConfig(): CameraXConfig = Camera2Config.defaultConfig() }
66. !!/** * Convenience class for generating a pre-populated Camera2 {@link CameraXConfig}. !*/ public final class Camera2Config { private Camera2Config() { } !!/** * Creates a {@link CameraXConfig} containing the default Camera2 implementation for CameraX. !*/ @NonNull public static CameraXConfig defaultConfig() { "// Create the camera factory for creating Camera2 camera objects CameraFactory.Provider cameraFactoryProvider = Camera2CameraFactory"::new; "// Create the DeviceSurfaceManager for Camera2 CameraDeviceSurfaceManager.Provider surfaceManagerProvider = Camera2DeviceSurfaceManager"::new; "// Create default configuration factory UseCaseConfigFactory.Provider configFactoryProvider = context "-> { ExtendableUseCaseConfigFactory factory = new ExtendableUseCaseConfigFactory(); factory.installDefaultProvider( ImageAnalysisConfig.class, new ImageAnalysisConfigProvider(context));
67. CameraDeviceSurfaceManager.Provider surfaceManagerProvider = Camera2DeviceSurfaceManager"::new; "// Create default configuration factory UseCaseConfigFactory.Provider configFactoryProvider = context "-> { ExtendableUseCaseConfigFactory factory = new ExtendableUseCaseConfigFactory(); factory.installDefaultProvider( ImageAnalysisConfig.class, new ImageAnalysisConfigProvider(context)); factory.installDefaultProvider( ImageCaptureConfig.class, new ImageCaptureConfigProvider(context)); factory.installDefaultProvider( VideoCaptureConfig.class, new VideoCaptureConfigProvider(context)); factory.installDefaultProvider( PreviewConfig.class, new PreviewConfigProvider(context)); return factory; }; CameraXConfig.Builder appConfigBuilder = new CameraXConfig.Builder() .setCameraFactoryProvider(cameraFactoryProvider) .setDeviceSurfaceManagerProvider(surfaceManagerProvider) .setUseCaseConfigFactoryProvider(configFactoryProvider); return appConfigBuilder.build(); } }
68. class XCameraView @JvmOverloads constructor( context: Context, attrs: AttributeSet? = null ) : PreviewView(context, attrs), BarcodeProcessor { private var cameraProviderFuture: ListenableFuture<ProcessCameraProvider> = ProcessCameraProvider.getInstance(context) private val cameraSelector: CameraSelector = CameraSelector.Builder() .requireLensFacing(CameraSelector.LENS_FACING_BACK) .build() init { cameraProviderFuture.addListener( Runnable { bindToLifecycle(cameraProviderFuture.get()) }, ContextCompat.getMainExecutor(context) ) }
69. public Camera bindToLifecycle(@NonNull LifecycleOwner lifecycleOwner, @NonNull CameraSelector cameraSelector, @NonNull UseCase""... useCases) { return CameraX.bindToLifecycle(lifecycleOwner, cameraSelector, useCases); }
70. private fun bindToLifecycle(cameraProvider: ProcessCameraProvider) { val preview: Preview = Preview.Builder() .setTargetName("Preview") .build() preview.previewSurfaceProvider = previewSurfaceProvider cameraProvider.bindToLifecycle( context as LifecycleOwner, cameraSelector, preview ) }
71. ML Kit's barcode scanning API •Reads most standard formats •Automatic format detection •Extracts structured data •Works with any orientation •Runs on the device
72. Version 24.0.0 of firebase-ml- vision introduces a new barcode scanning model
73. ML Kit is still in Beta ML Kit still requires a Firebase project
74. class FirebaseImageAnalyzer : BarcodeAnalyzer { private val onBarcodeScannedPublisher: PublishSubject<BarcodeModel> = PublishSubject.create() override fun onBarcodeScanned(): Observable<BarcodeModel> = onBarcodeScannedPublisher
75. class FirebaseImageAnalyzer : BarcodeAnalyzer { private val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() private val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) private val onBarcodeScannedPublisher: PublishSubject<BarcodeModel> = PublishSubject.create() override fun onBarcodeScanned(): Observable<BarcodeModel> = onBarcodeScannedPublisher
76. class FirebaseImageAnalyzer : ImageAnalysis.Analyzer, BarcodeAnalyzer { private val options = FirebaseVisionBarcodeDetectorOptions.Builder() .setBarcodeFormats(FirebaseVisionBarcode.FORMAT_ALL_FORMATS) .build() private val detector = FirebaseVision.getInstance() .getVisionBarcodeDetector(options) private val onBarcodeScannedPublisher: PublishSubject<BarcodeModel> = PublishSubject.create() override fun analyze(image: ImageProxy) { } override fun onBarcodeScanned(): Observable<BarcodeModel> = onBarcodeScannedPublisher private fun degreesToFirebaseRotation(degrees: Int): Int = when (degrees) { 0 "-> FirebaseVisionImageMetadata.ROTATION_0 90 "-> FirebaseVisionImageMetadata.ROTATION_90 180 "-> FirebaseVisionImageMetadata.ROTATION_180 270 "-> FirebaseVisionImageMetadata.ROTATION_270 else "-> throw Exception("Rotation must be 0, 90, 180, or 270.") }
77. private fun bindToLifecycle(cameraProvider: ProcessCameraProvider) { val preview: Preview = Preview.Builder() .setTargetName("Preview") .build() preview.previewSurfaceProvider = previewSurfaceProvider val imageAnalysis = ImageAnalysis.Builder() .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST) .setTargetResolution((Size(1280, 720))) .build() imageAnalysis.setAnalyzer( Executors.newSingleThreadExecutor(), firebaseImageAnalyzer ) cameraProvider.bindToLifecycle( context as LifecycleOwner, cameraSelector, preview, imageAnalysis ) }
78. override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
79. override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
80. override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
81. override fun analyze(image: ImageProxy) { val mediaImage = image.image val imageRotation = degreesToFirebaseRotation(image.imageInfo.rotationDegrees) if (mediaImage "!= null) { val firebaseVisionImage = FirebaseVisionImage.fromMediaImage(mediaImage, imageRotation) detector.detectInImage(firebaseVisionImage) .addOnSuccessListener { barcodes "-> if (barcodes.isEmpty()) return@addOnSuccessListener for (barcode in barcodes) { barcode.displayValue"?.let { value "-> val model = BarcodeModel( value = value, symbology = barcodeFormatToSymbology(barcode.format) ) onBarcodeScannedPublisher.onNext(model) } } } .addOnFailureListener { onBarcodeScannedPublisher.onError(it) } } image.close() }
82. !!/** * Only deliver the latest image to the analyzer, dropping images as they arrive. * * <p>This strategy ignores the value set by {@link Builder#setImageQueueDepth(int)}. * Only one image will be delivered for analysis at a time. If more images are produced * while that image is being analyzed, they will be dropped and not queued for delivery. * Once the image being analyzed is closed by calling {@link ImageProxy#close()}, the * next latest image will be delivered. * * <p>Internally this strategy may make use of an internal {@link Executor} to receive * and drop images from the producer. A performance-tuned executor will be created * internally unless one is explicitly provided by * {@link Builder#setBackgroundExecutor(Executor)}. In order to * ensure smooth operation of this backpressure strategy, any user supplied * {@link Executor} must be able to quickly respond to tasks posted to it, so setting * the executor manually should only be considered in advanced use cases. * * @see Builder#setBackgroundExecutor(Executor) !*/ public static final int STRATEGY_KEEP_ONLY_LATEST = 0;
83. class ScanActivity : AppCompatActivity() { companion object { const val VIBRATION_DURATION: Long = 250 } private val compositeDisposable = CompositeDisposable() override fun onCreate(savedInstanceState: Bundle?) { super.onCreate(savedInstanceState) setContentView(R.layout.activity_main) compositeDisposable.add(previewView.onBarcodeScanned() .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .subscribe( { vibrate() }, { Log.e("ScanActivity", "Scanning error", it) } ) ) }
84. <?xml version="1.0" encoding="utf-8"?> <androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http:"// schemas.android.com/apk/res/android" xmlns:app="http:"//schemas.android.com/apk/res-auto" xmlns:tools="http:"//schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" tools:context=".ScanActivity"> <com.alephom.android.scan.views.XCameraView android:id="@+id/previewView" android:layout_width="wrap_content" android:layout_height="wrap_content" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent" "/> "</androidx.constraintlayout.widget.ConstraintLayout>
85. <?xml version="1.0" encoding="utf-8"?> <androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http:"// schemas.android.com/apk/res/android" xmlns:app="http:"//schemas.android.com/apk/res-auto" xmlns:tools="http:"//schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" tools:context=".ScanActivity"> <com.alephom.android.scan.views.OCameraView android:id="@+id/previewView" android:layout_width="wrap_content" android:layout_height="wrap_content" app:cameraAudio="off" app:layout_constraintBottom_toBottomOf="parent" app:layout_constraintEnd_toEndOf="parent" app:layout_constraintStart_toStartOf="parent" app:layout_constraintTop_toTopOf="parent" "/> "</androidx.constraintlayout.widget.ConstraintLayout>
86. Go beyond…
87. Android Dev Summit ’19 https:"//www.youtube.com/watch?v=CSJMxEOgiJQ
88. • Barcode is far away • Barcode is distorted source: Android Dev Summit ’19
89. source: Android Dev Summit ’19
90. source: Android Dev Summit ’19
91. source: Android Dev Summit ’19
92. source: Android Dev Summit ’19
93. Machine Learning?
96. Reading barcodes with neural networks Department of Electrical Engineering Linköping University SE-581 83 Linköping, Sweden Copyright © 2017 Fredrik Fridborn
105. Thank you