Machine Learning for Android Developers Moyinoluwa Adeyemi November 25, 2016

Android @SwiftaSystems GDG Lagos Co-organizer Android Developer Nanodegree Graduate Half Marathoner

How Can You Get Started with Machine Learning? Three ways, with varying complexity: (1) Use a Cloud-based or Mobile API (Vision, Natural Language, etc.) (2) Use an existing model architecture, and retrain it or fine tune on your dataset (3) Develop your own machine learning models for new problems More flexible, but more effort required

Mobile Vision API Providing on-device vision for applications

Face API faces, facial landmarks, eyes open, smiling Barcode API 1D and 2D barcodes Text API Latin-based text / structure Common Mobile Vision API Support for fast image and video on-device detection and tracking. NEW!

Googly Eyes Android App Video credit Google 1. Create a face detector for facial landmarks (e.g., eyes) 3. For each face, draw the eyes FaceDetector detector = new FaceDetector.Builder() .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build(); SparseArray faces = detector.detect(image); for (int i = 0; i < faces.size(); ++i) { Face face = faces.valueAt(i); for (Landmark landmark : face.getLandmarks()) { // Draw eyes 2. Detect faces in the image

Face API Photo credit developers.google.com/vision

Easy to use Java API image detected items Detector 1. Create a detector object 2. detectedItems = detector.detect(image) Photo credit developers.google.com/vision

Text Detection Latin based language Understand text structure Photo credit Getty Images

Text Structure Blocks Lines Words Lines Words Words Words

Barcode Detection 1D barcodes EAN-13/8 UPC-A/E Code-39/93/128 ITF Codabar 2D barcodes QR Code Data Matrix PDF-417 AZTEC UPC DataMatrix QR Code PDF 417 Video and image credit Google

Combined Vision & Translation

Code Walkthrough

Import the Google Play Services SDK for the Mobile Vision API compile 'com.google.android.gms:play-services-vision:9.8.0'

Face Detection manifest file meta-data

Barcode Detection manifest file meta-data

Text Recognition manifest file meta-data

Create simple layout with static image Process

Convert the static image into a Bitmap Bitmap barcodeBitmap = BitmapFactory.decodeResource(getResources(), R.drawable.cute_cat_image);

Using the FaceDetector API FaceDetector faceDetector = new FaceDetector.Builder(this) .setTrackingEnabled(false) .setLandmarkType(FaceDetector.ALL_LANDMARKS) .build();

Using the BarcodeDetector API BarcodeDetector barcodeDetector = new BarcodeDetector.Builder(this) .setBarcodeFormats(Barcode.ALL_FORMATS) .build();

Using the TextRecognizer API TextRecognizer textRecognizer = new TextRecognizer.Builder(this) .build();

Check if the face detector is ready if (!faceDetector.isOperational()) { new AlertDialog.Builder(this) .setMessage("Face Detector could not be set up on your device") .show(); return; }

Check if the barcode detector is ready if (!barcodeDetector.isOperational()) { new AlertDialog.Builder(this) .setMessage("Barcode Detector could not be set up on your device") .show(); return; }

Check if the text recognizer is ready if (!textRecognizer.isOperational()) { new AlertDialog.Builder(this) .setMessage("Text Recognizer could not be set up on your device") .show(); return; }

Create a frame using the face bitmap Frame frame = new Frame.Builder().setBitmap(textBitmap).build(); SparseArray face = faceDetector.detect(frame);

Create a frame using the barcode bitmap Frame frame = new Frame.Builder().setBitmap(textBitmap).build(); SparseArray barcode = barcodeDetector.detect(frame);

Create a frame using the text bitmap Frame frame = new Frame.Builder().setBitmap(textBitmap).build(); SparseArray text = textRecognizer.detect(frame);

Use the values from the SparseArray for (int i = 0; i < sparseArray.size(); i++) { Face face = sparseArray.valueAt(i); float left = face.getPosition().x; float top = face.getPosition().y; float right = left + face.getWidth(); float bottom = right + face.getHeight(); float cornerRadius = 2.0f; RectF rectF = new RectF(left, top, right, bottom); canvas.drawRoundRect(rectF, cornerRadius, cornerRadius, rectPaint); }

Use the values from the SparseArray for (int i = 0; i < sparseArray.size(); i++) { Face face = sparseArray.valueAt(i); float left = face.getPosition().x; float top = face.getPosition().y; float right = left + face.getWidth(); float bottom = right + face.getHeight(); float cornerRadius = 2.0f; RectF rectF = new RectF(left, top, right, bottom); canvas.drawRoundRect(rectF, cornerRadius, cornerRadius, rectPaint); }

Use the values from the SparseArray if (size == 0) { textView.setText("No information available"); } else { for (int i = 0; i < size; i++) { barcodeValue += (barcode.valueAt(i).displayValue + "\n"); } textView.setText(barcodeValue); }

for (int i = 0; i < text.size(); i++) { TextBlock textBlock = text.valueAt(i); if (textBlock != null && textBlock.getValue() != null) { detectedText += textBlock.getValue(); } detectedTextView.setText(detectedText); } Use the values from the SparseArray

faceDetector.release(); Remember to release resources

barcodeDetector.release(); Remember to release resources

textRecognizer.release(); Remember to release resources

Results - Face Detection API

Results - Barcode Detection API

Results - Text Recognition API

Mobile Vision: Codelabs and Samples Googly Eyes Code Sample github.com/googlesamples/android-vision/tree/master/visionSamples/googly-eyes Codelabs codelabs.developers.google.com/codelabs/face-detection/ codelabs.developers.google.com/codelabs/bar-codes/ codelabs.developers.google.com/codelabs/mobile-vision-ocr/ Mobile Vision Developers developers.google.com/vision/ GitHub Code Samples github.com/googlesamples/android-vision Stack Overflow Find and ask questions under the android-vision tag.

Questions?

@moyheen @moyinoluwa @moyheen Thank you! Moyinoluwa Adeyemi Android @SwiftaSystems GDG Lagos co-organizer