Seeing is Believing: Mobile Vision API Deep Dive

Seeing is Believing: Mobile Vision API Deep Dive

Did you know that your mobile device can recognize objects in photos and videos? It can detect faces, barcodes and text in an image and even detect whether those faces are smiling or winking! All this is possible because of the Mobile Vision API which finds objects in photos and videos on mobile devices, using real-time on-device vision technology. This API will enable you edit photos and videos, automate text data entry, embed barcode scanners into your applications and even build your own version of a Stories application with lots of filters. In this talk, you’ll learn how to use the Mobile Vision API to get more out of your photos and videos. You’ll get a jumpstart on implementing Machine Learning in your apps with the power of this API!

E803718649600ddffc1bc625d957e786?s=128

Moyinoluwa Adeyemi

September 25, 2017
Tweet

Transcript

  1. 4.

    • Finds objects in photos and videos • Includes face,

    barcode and text detectors Mobile Vision API
  2. 5.

    • Finds objects in photos and videos • Includes face,

    barcode and text detectors • Easy to set up Mobile Vision API
  3. 6.

    • Finds objects in photos and videos • Includes face,

    barcode and text detectors • Easy to set up • Works locally Mobile Vision API
  4. 7.

    • Finds objects in photos and videos • Includes face,

    barcode and text detectors • Easy to set up • Works locally • Available offline Mobile Vision API
  5. 8.

    • Finds objects in photos and videos • Includes face,

    barcode and text detectors • Easy to set up • Works locally • Available offline • Free Mobile Vision API
  6. 9.
  7. 12.

    Euler X - up/down Euler Y - left/right Euler Z

    - rotated/slated Understands faces positioned at different angles https://developers.google.com/vision/face-detection-concepts
  8. 13.

    https://pixabay.com/en/woman-stylish-fashion-view-101542/ Detects landmarks Left eye - 4 Right eye -

    10 Nose base - 6 Left cheek - 1 Right cheek - 7 Left, Right and Bottom Mouth - 5, 11, 0
  9. 26.

    Spanish English Hungarian Norwegian German Dutch French Catalan Portugese Romanian

    Polish Danish Finnish Italian Swedish Turkish Works only on Latin based languages
  10. 29.
  11. 30.
  12. 31.
  13. 35.

    <uses-feature android:name="android.hardware.camera" /> <uses-permission android:name="android.permission.CAMERA" /> <meta-data android:name="com.google.android.gms.vision.DEPENDENCIES" android:value="face, barcode,

    text" /> <meta-data android:name="com.google.android.gms.version" android:value="@integer/google_play_services_version"/> Update the manifest file
  14. 40.

    Convert photo to a mutable bitmap val bitmapOptions = BitmapFactory.Options().apply

    { inMutable = true } val bitmap = BitmapFactory.decodeResource(resources, R.drawable.image, bitmapOptions)
  15. 42.

    val tempBitmap = Bitmap.createBitmap(bitmap.width, bitmap.height, Bitmap.Config.RGB_565) val paint = Paint().apply

    { strokeWidth = 5f color = Color.MAGENTA style = Paint.Style.STROKE } Prepare the canvas
  16. 43.

    val tempBitmap = Bitmap.createBitmap(bitmap.width, bitmap.height, Bitmap.Config.RGB_565) val paint = Paint().apply

    { strokeWidth = 5f color = Color.MAGENTA style = Paint.Style.STROKE } val canvas = Canvas(tempBitmap) canvas.drawBitmap(bitmap, 0f, 0f, paint) Prepare the canvas
  17. 51.

    if (!faceDetector.isOperational) { toast("Face Detector could not be set up

    on your device :(") return } Confirm the operationality
  18. 53.

    Calculate coordinates for (i in 0 until faceArray.size()) { val

    face = faceArray.valueAt(i) val left = face.position.x val top = face.position.y val right = left + face.width val bottom = top + face.height } https://pixabay.com/en/woman-stylish-fashion-view-101542/ (x, y) left, top (x + width, y + height) right, bottom
  19. 54.

    Draw graphics on face for (i in 0 until faceArray.size())

    { val face = faceArray.valueAt(i) ... val bound = RectF(left, top, right, a bottom) canvas.drawRoundRect(bound, c cornerRadius, cornerRadius, paint) } https://pixabay.com/en/woman-stylish-fashion-view-101542/
  20. 55.

    Draw on all landmarks for (landmark in face.landmarks) { val

    x = landmark.position.x val y = landmark.position.y val landmarkType = landmark.type.toString() C canvas.drawText(landmarkType, x, y, p paint) } https://pixabay.com/en/woman-stylish-fashion-view-101542/
  21. 56.

    Draw on a specific landmark for (landmark in face.landmarks) {

    val x = landmark.position.x val y = landmark.position.y when (landmark.type) { 1, 7 -> canvas.drawCircle(x, y, r radius, paint) } } https://pixabay.com/en/woman-stylish-fashion-view-101542/
  22. 57.

    Detect classification for (i in 0 until faceArray.size()) { val

    face = faceArray.valueAt(i) // face.isSmilingProbability // face.isLeftEyeOpenProbability // face.isRightEyeOpenProbability when (face.isSmilingProbability) { in 0.0f..0.49f -> // TODO in 0.5f..1f -> payForFriedChicken() } ... https://pixabay.com/en/woman-stylish-fashion-view-101542/
  23. 58.

    Detect position for (i in 0 until faceArray.size()) { val

    face = faceArray.valueAt(i) // face.eulerY // face.eulerZ Log.i(TAG, face.eulerY.toString()) Log.i(TAG, face.eulerZ.toString()) } https://pixabay.com/en/woman-stylish-fashion-view-101542/
  24. 61.

    Create an immutable bitmap (since we don’t intend to draw

    on it) val bitmap = BitmapFactory.decodeResource(resources, R.drawable.image)
  25. 67.

    Retrieve info - barcode for (i in 0 until barcodeArray.size())

    { val barcodeData = barcodeArray.valueAt(i) }
  26. 68.

    Retrieve info - barcode for (i in 0 until barcodeArray.size())

    { val barcodeData = barcodeArray.valueAt(i).wifi }
  27. 69.

    Retrieve info - text for (i in 0 until barcodeArray.size())

    { val barcodeData = barcodeArray.valueAt(i).wifi } (0 until textArray.size()) .map { textArray.valueAt(it) } .filterNot { it.value.isNullOrBlank() } .forEach { println(it.value) }
  28. 70.

    Retrieve info - text for (i in 0 until barcodeArray.size())

    { val barcodeData = barcodeArray.valueAt(i).wifi } (0 until textArray.size()) .map { textArray.valueAt(it) } .filterNot { it.value.isNullOrBlank() } .forEach { println(it.value) }
  29. 71.

    Retrieve info - text for (i in 0 until barcodeArray.size())

    { val barcodeData = barcodeArray.valueAt(i).wifi } (0 until textArray.size()) .map { textArray.valueAt(it) } .filterNot { it.value.isNullOrBlank() } .forEach { println(it.value) }
  30. 72.

    Retrieve info - text for (i in 0 until barcodeArray.size())

    { val barcodeData = barcodeArray.valueAt(i).wifi } (0 until textArray.size()) .map { textArray.valueAt(it) } .filterNot { it.value.isNullOrBlank() } .forEach { println(it.value) }
  31. 75.
  32. 78.

    Set up a face tracker class FaceTracker internal constructor( private

    val overlay: GraphicOverlay ) : Tracker<Face>() { private val faceGraphic: FaceGraphic = FaceGraphic(overlay) ... }
  33. 79.

    Set up a face tracker override fun onNewItem override fun

    onUpdate override fun onMissing override fun onDone
  34. 80.

    Set up a face tracker factory class FaceTrackerFactory : MultiProcessor.Factory<Face>

    { override fun create(face: Face): Tracker<Face> = FaceTracker(overlay) }
  35. 81.

    Set up a face detector val detector = FaceDetector.Builder(this) .setTrackingEnabled(true)...

    detector.setProcessor( MultiProcessor.Builder(faceTrackerFactory).build()) if (!detector.isOperational) { //... }
  36. 82.

    Create the camera source cameraSource = CameraSource.Builder(this, detector) .setRequestedPreviewSize(640, 480)

    .setFacing(CameraSource.CAMERA_FACING_BACK) .setRequestedFps(30.0f) .build()
  37. 85.

    Update the UI • Include a CameraPreview • Include the

    overlay classes - FaceGraphic.kt, BarcodeGraphic.kt, TextGraphic.kt
  38. 86.

    Set up the respective object trackers class FaceTracker internal constructor(

    private val overlay: GraphicOverlay ) : Tracker<Face>() { private val faceGraphic: FaceGraphic = FaceGraphic(overlay) ... }
  39. 87.

    Set up the respective object trackers class BarcodeTracker internal constructor(

    private val overlay: GraphicOverlay ) : Tracker<Barcode>() { private val barcodeGraphic: BarcodeGraphic = BarcodeGraphic(overlay) ... }
  40. 88.

    As well as the tracker factories class FaceTrackerFactory : MultiProcessor.Factory<Face>

    { override fun create(face: Face): Tracker<Face> = FaceTracker(overlay) } class BarcodeTrackerFactory : MultiProcessor.Factory<Barcode> { override fun create(barcode: Barcode): Tracker<Barcode> = BarcodeTracker(overlay) }
  41. 90.

    Set up the detectors val faceDetector = FaceDetector.Builder(this).build() val barcodeDetector

    = BarcodeDetector.Builder(this).build() faceDetector.setProcessor( MultiProcessor.Builder(faceTrackerFactory).build()) barcodeDetector.setProcessor( MultiProcessor.Builder(barcodeTrackerFactory).build())
  42. 91.

    Wrap the detectors with a MultiDetector val multiDetector = MultiDetector.Builder()

    .add(faceDetector) .add(barcodeDetector) .build() if (!multiDetector.isOperational) { //... }
  43. 92.

    Create the camera source cameraSource = CameraSource.Builder(this, multiDetector) .setRequestedPreviewSize(640, 480)

    .setFacing(CameraSource.CAMERA_FACING_BACK) .setRequestedFps(30.0f) .build()