Seeing is Believing: Mobile Vision API Deep Dive

Slide 1

Slide 1 text

Seeing is Believing Mobile Vision API Deep Dive

Slide 2

Slide 2 text

Moyinoluwa Adeyemi Off Grid Electric @moyheen

Slide 3

Slide 3 text

Mobile Vision API ● Finds objects in photos and videos

Slide 4

Slide 4 text

● Finds objects in photos and videos ● Includes face, barcode and text detectors Mobile Vision API

Slide 5

Slide 5 text

● Finds objects in photos and videos ● Includes face, barcode and text detectors ● Easy to set up Mobile Vision API

Slide 6

Slide 6 text

● Finds objects in photos and videos ● Includes face, barcode and text detectors ● Easy to set up ● Works locally Mobile Vision API

Slide 7

Slide 7 text

● Finds objects in photos and videos ● Includes face, barcode and text detectors ● Easy to set up ● Works locally ● Available offline Mobile Vision API

Slide 8

Slide 8 text

● Finds objects in photos and videos ● Includes face, barcode and text detectors ● Easy to set up ● Works locally ● Available offline ● Free Mobile Vision API

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Face Detection

Slide 11

Slide 11 text

Detects human faces Sorry dawg, humans only.

Slide 12

Slide 12 text

Euler X - up/down Euler Y - left/right Euler Z - rotated/slated Understands faces positioned at different angles https://developers.google.com/vision/face-detection-concepts

Slide 13

Slide 13 text

https://pixabay.com/en/woman-stylish-fashion-view-101542/ Detects landmarks Left eye - 4 Right eye - 10 Nose base - 6 Left cheek - 1 Right cheek - 7 Left, Right and Bottom Mouth - 5, 11, 0

Slide 14

Slide 14 text

https://pixabay.com/en/woman-stylish-fashion-view-101542/ Understands facial expressions isSmilingProbability: 0.006698033

Slide 15

Slide 15 text

https://pixabay.com/en/woman-stylish-fashion-view-101542/ Understands facial expressions isSmilingProbability: 0.006698033 isLeftEyeOpenProbability: 0.98714304

Slide 16

Slide 16 text

https://pixabay.com/en/woman-stylish-fashion-view-101542/ Understands facial expressions isSmilingProbability: 0.006698033 isLeftEyeOpenProbability: 0.98714304 isRightEyeOpenProbability: 0.69178355

Slide 17

Slide 17 text

Works on all skin colors

Slide 18

Slide 18 text

Barcode Detection

Slide 19

Slide 19 text

Detects barcodes in images and videos

Slide 20

Slide 20 text

Works on both 1D and 2D barcodes https://www.adazonusa.com/blog/wp-content/uploads/2016/03/1D-barcode-vs-2D-barcodes.jpg

Slide 21

Slide 21 text

Detects multiple barcodes in one image

Slide 22

Slide 22 text

Even when they are upside down

Slide 23

Slide 23 text

Text Detection

Slide 24

Slide 24 text

Detects texts in image and video #TIL

Slide 25

Slide 25 text

Segments text into block, lines and words https://developers.google.com/vision/images/text-structure.png

Slide 26

Slide 26 text

Spanish English Hungarian Norwegian German Dutch French Catalan Portugese Romanian Polish Danish Finnish Italian Swedish Turkish Works only on Latin based languages

Slide 27

Slide 27 text

Has a wide range of applications

Slide 28

Slide 28 text

Getting Started

Slide 29

Slide 29 text

No content

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

No content

Slide 32

Slide 32 text

dependencies { compile 'com.google.android.gms:play-services-vision:11.2.2' } Update the gradle file

Slide 33

Slide 33 text

Update the manifest file

Slide 34

Slide 34 text

Update the manifest file

Slide 35

Slide 35 text

Update the manifest file

Slide 36

Slide 36 text

Run only on a background thread CAUTION DO NOT RUN ON UI THREAD

Slide 37

Slide 37 text

faceDetector.release() barcodeDetector.release() textRecognizer.release() Release resources at the end

Slide 38

Slide 38 text

GIVEN any photo WHEN any face is detected THEN overlay image and graphics

Slide 39

Slide 39 text

Convert photo to a mutable bitmap val bitmapOptions = BitmapFactory.Options().apply { inMutable = true }

Slide 40

Slide 40 text

Convert photo to a mutable bitmap val bitmapOptions = BitmapFactory.Options().apply { inMutable = true } val bitmap = BitmapFactory.decodeResource(resources, R.drawable.image, bitmapOptions)

Slide 41

Slide 41 text

val tempBitmap = Bitmap.createBitmap(bitmap.width, bitmap.height, Bitmap.Config.RGB_565) Prepare the canvas

Slide 42

Slide 42 text

val tempBitmap = Bitmap.createBitmap(bitmap.width, bitmap.height, Bitmap.Config.RGB_565) val paint = Paint().apply { strokeWidth = 5f color = Color.MAGENTA style = Paint.Style.STROKE } Prepare the canvas

Slide 43

Slide 43 text

Slide 44

Slide 44 text

val faceDetector = FaceDetector.Builder(this) .setClassificationType(FaceDetector.ALL_CLASSIFICATIONS) .setTrackingEnabled(false) .setProminentFaceOnly(false) .setLandmarkType(FaceDetector.ALL_LANDMARKS) .setMode(FaceDetector.ACCURATE_MODE) .setMinFaceSize(0.2f) .build() Set up the face detector

Slide 45

Slide 45 text

Slide 46

Slide 46 text

Slide 47

Slide 47 text

Slide 48

Slide 48 text

Slide 49

Slide 49 text

Slide 50

Slide 50 text

Slide 51

Slide 51 text

if (!faceDetector.isOperational) { toast("Face Detector could not be set up on your device :(") return } Confirm the operationality

Slide 52

Slide 52 text

val frame = Frame.Builder().setBitmap(bitmap).build() val faceArray = faceDetector.detect(frame) Detect faces

Slide 53

Slide 53 text

Calculate coordinates for (i in 0 until faceArray.size()) { val face = faceArray.valueAt(i) val left = face.position.x val top = face.position.y val right = left + face.width val bottom = top + face.height } https://pixabay.com/en/woman-stylish-fashion-view-101542/ (x, y) left, top (x + width, y + height) right, bottom

Slide 54

Slide 54 text

Draw graphics on face for (i in 0 until faceArray.size()) { val face = faceArray.valueAt(i) ... val bound = RectF(left, top, right, a bottom) canvas.drawRoundRect(bound, c cornerRadius, cornerRadius, paint) } https://pixabay.com/en/woman-stylish-fashion-view-101542/

Slide 55

Slide 55 text

Draw on all landmarks for (landmark in face.landmarks) { val x = landmark.position.x val y = landmark.position.y val landmarkType = landmark.type.toString() C canvas.drawText(landmarkType, x, y, p paint) } https://pixabay.com/en/woman-stylish-fashion-view-101542/

Slide 56

Slide 56 text

Draw on a specific landmark for (landmark in face.landmarks) { val x = landmark.position.x val y = landmark.position.y when (landmark.type) { 1, 7 -> canvas.drawCircle(x, y, r radius, paint) } } https://pixabay.com/en/woman-stylish-fashion-view-101542/

Slide 57

Slide 57 text

Detect classification for (i in 0 until faceArray.size()) { val face = faceArray.valueAt(i) // face.isSmilingProbability // face.isLeftEyeOpenProbability // face.isRightEyeOpenProbability when (face.isSmilingProbability) { in 0.0f..0.49f -> // TODO in 0.5f..1f -> payForFriedChicken() } ... https://pixabay.com/en/woman-stylish-fashion-view-101542/

Slide 58

Slide 58 text

Detect position for (i in 0 until faceArray.size()) { val face = faceArray.valueAt(i) // face.eulerY // face.eulerZ Log.i(TAG, face.eulerY.toString()) Log.i(TAG, face.eulerZ.toString()) } https://pixabay.com/en/woman-stylish-fashion-view-101542/

Slide 59

Slide 59 text

Update the image and release resources imageView.setImageDrawable(BitmapDrawable(resources, tempBitmap)) faceDetector.release()

Slide 60

Slide 60 text

GIVEN any photo WHEN any barcode or text is detected THEN retrieve information

Slide 61

Slide 61 text

Create an immutable bitmap (since we don’t intend to draw on it) val bitmap = BitmapFactory.decodeResource(resources, R.drawable.image)

Slide 62

Slide 62 text

val barcodeDetector = BarcodeDetector.Builder(this) .setBarcodeFormats(Barcode.ALL_FORMATS) .build() Set up the detectors

Slide 63

Slide 63 text

val barcodeDetector = BarcodeDetector.Builder(this) .setBarcodeFormats(Barcode.ALL_FORMATS) .build() Set up the detectors

Slide 64

Slide 64 text

val barcodeDetector = BarcodeDetector.Builder(this) .setBarcodeFormats(Barcode.ALL_FORMATS) .build() val textRecognizer = TextRecognizer.Builder(this).build() Set up the detectors

Slide 65

Slide 65 text

if (!barcodeDetector.isOperational || !textRecognizer.isOperational) { return } Confirm the operationality

Slide 66

Slide 66 text

val frame = Frame.Builder().setBitmap(bitmap).build() val barcodeArray = barcodeDetector.detect(frame) val textArray = textRecognizer.detect(frame) Detect barcodes and text

Slide 67

Slide 67 text

Retrieve info - barcode for (i in 0 until barcodeArray.size()) { val barcodeData = barcodeArray.valueAt(i) }

Slide 68

Slide 68 text

Retrieve info - barcode for (i in 0 until barcodeArray.size()) { val barcodeData = barcodeArray.valueAt(i).wifi }

Slide 69

Slide 69 text

Retrieve info - text for (i in 0 until barcodeArray.size()) { val barcodeData = barcodeArray.valueAt(i).wifi } (0 until textArray.size()) .map { textArray.valueAt(it) } .filterNot { it.value.isNullOrBlank() } .forEach { println(it.value) }

Slide 70

Slide 70 text

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Slide 73

Slide 73 text

Release resources barcodeDetector.release() textRecognizer.release()

Slide 74

Slide 74 text

GIVEN any back camera WHEN any face is detected THEN overlay graphics and images

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

Update the UI ● Include a CameraPreview

Slide 77

Slide 77 text

Update the UI ● Include a CameraPreview ● Include an overlay class

Slide 78

Slide 78 text

Set up a face tracker class FaceTracker internal constructor( private val overlay: GraphicOverlay ) : Tracker() { private val faceGraphic: FaceGraphic = FaceGraphic(overlay) ... }

Slide 79

Slide 79 text

Set up a face tracker override fun onNewItem override fun onUpdate override fun onMissing override fun onDone

Slide 80

Slide 80 text

Set up a face tracker factory class FaceTrackerFactory : MultiProcessor.Factory { override fun create(face: Face): Tracker = FaceTracker(overlay) }

Slide 81

Slide 81 text

Set up a face detector val detector = FaceDetector.Builder(this) .setTrackingEnabled(true)... detector.setProcessor( MultiProcessor.Builder(faceTrackerFactory).build()) if (!detector.isOperational) { //... }

Slide 82

Slide 82 text

Create the camera source cameraSource = CameraSource.Builder(this, detector) .setRequestedPreviewSize(640, 480) .setFacing(CameraSource.CAMERA_FACING_BACK) .setRequestedFps(30.0f) .build()

Slide 83

Slide 83 text

GIVEN any back camera WHEN any object is detected THEN overlay graphics and images

Slide 84

Slide 84 text

Update the UI ● Include a CameraPreview

Slide 85

Slide 85 text

Update the UI ● Include a CameraPreview ● Include the overlay classes - FaceGraphic.kt, BarcodeGraphic.kt, TextGraphic.kt

Slide 86

Slide 86 text

Set up the respective object trackers class FaceTracker internal constructor( private val overlay: GraphicOverlay ) : Tracker() { private val faceGraphic: FaceGraphic = FaceGraphic(overlay) ... }

Slide 87

Slide 87 text

Set up the respective object trackers class BarcodeTracker internal constructor( private val overlay: GraphicOverlay ) : Tracker() { private val barcodeGraphic: BarcodeGraphic = BarcodeGraphic(overlay) ... }

Slide 88

Slide 88 text

As well as the tracker factories class FaceTrackerFactory : MultiProcessor.Factory { override fun create(face: Face): Tracker = FaceTracker(overlay) } class BarcodeTrackerFactory : MultiProcessor.Factory { override fun create(barcode: Barcode): Tracker = BarcodeTracker(overlay) }

Slide 89

Slide 89 text

Set up the detectors val faceDetector = FaceDetector.Builder(this).build() val barcodeDetector = BarcodeDetector.Builder(this).build()

Slide 90

Slide 90 text

Set up the detectors val faceDetector = FaceDetector.Builder(this).build() val barcodeDetector = BarcodeDetector.Builder(this).build() faceDetector.setProcessor( MultiProcessor.Builder(faceTrackerFactory).build()) barcodeDetector.setProcessor( MultiProcessor.Builder(barcodeTrackerFactory).build())

Slide 91

Slide 91 text

Wrap the detectors with a MultiDetector val multiDetector = MultiDetector.Builder() .add(faceDetector) .add(barcodeDetector) .build() if (!multiDetector.isOperational) { //... }

Slide 92

Slide 92 text

Create the camera source cameraSource = CameraSource.Builder(this, multiDetector) .setRequestedPreviewSize(640, 480) .setFacing(CameraSource.CAMERA_FACING_BACK) .setRequestedFps(30.0f) .build()

Slide 93

Slide 93 text

References https://developers.google.com/vision/ https://codelabs.developers.google.com/codelabs/face-detection/ind ex.html https://github.com/googlesamples/android-vision/tree/master/vision Samples Illustrations The extremely talented Virginia Poltrack - @VPoltrack

Slide 94

Slide 94 text

Moyinoluwa Adeyemi Off Grid Electric @moyheen Thank you!