Machine Learning On Mobile With MLKit Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Can't we already do that? Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

https://www.tensorflow.org/lite/ 

Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

TensorFlow Lite API Reference Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

class TensorFlowLiteDemo(modelFile: File, labels: Array) { private val labelProbArray: Array = arrayOf(FloatArray(labels.size)) private val tflite: Interpreter = Interpreter(modelFile) fun classifyImage(bitmap: Bitmap) { val imageData = convertBitmapToByteBuffer(bitmap) tflite.run(imageData, labelProbArray) } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Developer Experience Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

On-device & Cloud Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Offline/On-device Machine Learning → Reduced accuracy → Larger downloads → Increased resource usage Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

MLKit on iOS - Setup pod 'Firebase/Core' pod 'Firebase/MLVision' # If using an on-device API: pod 'Firebase/MLVisionTextModel' Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Text recognition (iOS) let vision = Vision.vision() let textRecognizer = vision.onDeviceTextRecognizer() let image = VisionImage(image: uiImage) textRecognizer.process(visionImage) { result, error in guard error == nil, let result = result else { // ... return } // Recognized text } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

let resultText = result.text for block in result.blocks { let blockText = block.text let blockConfidence = block.confidence let blockLanguages = block.recognizedLanguages let blockCornerPoints = block.cornerPoints let blockFrame = block.frame for line in block.lines { let lineText = line.text let lineConfidence = line.confidence let lineLanguages = line.recognizedLanguages let lineCornerPoints = line.cornerPoints let lineFrame = line.frame for element in line.elements { let elementText = element.text let elementConfidence = element.confidence let elementLanguages = element.recognizedLanguages let elementCornerPoints = element.cornerPoints let elementFrame = element.frame } } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

MLKit on Android - Setup dependencies { // ... implementation 'com.google.firebase:firebase-ml-vision:18.0.1' implementation 'com.google.firebase:firebase-ml-vision-image-label-model:17.0.2' implementation 'com.google.firebase:firebase-ml-vision-face-model:17.0.2' } ... Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Text recognition (Android) val image = FirebaseVisionImage.fromBitmap(bitmap) val textRecognizer = FirebaseVision.instance.getOnDeviceTextRecognizer(); textRecognizer.processImage(image) .addOnSuccessListener({ result: FirebaseVisionText -> /* Task completed successfully */ }) .addOnFailureListener({ error: Exception -> /* Task failed with an exception */ }) Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

val resultText = result.getText result.getTextBlocks.forEach {block -> val blockText = block.text val blockConfidence = block.confidence val blockLanguages = block.recognizedLanguages val blockCornerPoints = block.cornerPoints val blockFrame = block.boundingBox block.lines.forEach { line -> val lineText = line.text val lineConfidence = line.confidence val lineLanguages = line.recognizedLanguages val lineCornerPoints = line.cornerPoints val lineFrame = line.boundingBox line.elements.forEach {element -> val elementText = element.text val elementConfidence = element.confidence val elementLanguages = element.recognizedLanguages val elementCornerPoints = element.cornerPoints() val elementFrame = element.boundingBox } } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Dealing with camera rotation object ORIENTATIONS: SparseIntArray() { init { append(Surface.ROTATION_0, 90) append(Surface.ROTATION_90, 0) append(Surface.ROTATION_180, 270) append(Surface.ROTATION_270, 180) } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

private fun calculateRotationCompensation(cameraId: String, activity: Activity): Int { val deviceRotation = activity.windowManager.defaultDisplay.rotation var rotationCompensation = ORIENTATIONS.get(deviceRotation) val cameraManager = activity.getSystemService(CAMERA_SERVICE) as CameraManager val sensorOrientation = cameraManager .getCameraCharacteristics(cameraId) .get(CameraCharacteristics.SENSOR_ORIENTATION)!! rotationCompensation = (rotationCompensation + sensorOrientation + 270) % 360 // Return the corresponding FirebaseVisionImageMetadata rotation value. return when (rotationCompensation) { 0 -> FirebaseVisionImageMetadata.ROTATION_0 90 -> FirebaseVisionImageMetadata.ROTATION_90 180 -> FirebaseVisionImageMetadata.ROTATION_180 270 -> FirebaseVisionImageMetadata.ROTATION_270 else -> { FirebaseVisionImageMetadata.ROTATION_0 } } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Face Detection Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Face Detection (Android) // High-accuracy landmark detection and face classification fun setupFacedetection(): FirebaseVisionFaceDetector { val options = FirebaseVisionFaceDetectorOptions.Builder() .setPerformanceMode(FirebaseVisionFaceDetectorOptions.ACCURATE) .setLandmarkMode(FirebaseVisionFaceDetectorOptions.ALL_LANDMARKS) .setClassificationMode(FirebaseVisionFaceDetectorOptions.ALL_CLASSIFICATIONS) .build() return FirebaseVision.getInstance().getVisionFaceDetector(options) } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

fun detectFaces(detector: FirebaseVisionFaceDetector, image: Bitmap) { val visionImage = FirebaseVisionImage.fromBitmap(image) detector.detectInImage(visionImage) .addOnSuccessListener { faces -> for (face in faces) { // Faces detected } } .addOnFailureListener { // TODO Error handling } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

val bounds = face.boundingBox val rotY = face.headEulerAngleY // Head is rotated to the right rotY degrees val rotZ = face.headEulerAngleZ // Head is tilted sideways rotZ degrees // If landmark detection was enabled (mouth, ears, eyes, cheeks, and // nose available): val leftEar = face.getLandmark(FirebaseVisionFaceLandmark.LEFT_EAR) if (leftEar != null) { val leftEarPos = leftEar.position } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

// If contour detection was enabled: val leftEyeContour = face.getContour(FirebaseVisionFaceContour.LEFT_EYE).points val upperLipBottomContour = face.getContour(FirebaseVisionFaceContour.UPPER_LIP_BOTTOM).points Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

// If classification was enabled: if (face.smilingProbability != FirebaseVisionFace.UNCOMPUTED_PROBABILITY) { val smileProb = face.smilingProbability } if (face.rightEyeOpenProbability != FirebaseVisionFace.UNCOMPUTED_PROBABILITY) { val rightEyeOpenProb = face.rightEyeOpenProbability } // If face tracking was enabled: if (face.trackingId != FirebaseVisionFace.INVALID_ID) { val id = face.trackingId } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Cloud Image Labeling (Android) fun createCloudLabelDetector(): FirebaseVisionCloudLabelDetector? { val options = FirebaseVisionCloudDetectorOptions.Builder() .setModelType(FirebaseVisionCloudDetectorOptions.LATEST_MODEL) .setMaxResults(15) .build() return FirebaseVision.getInstance().getVisionCloudLabelDetector(options) } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

fun labelImage(detector: FirebaseVisionCloudLabelDetector, bitmap: Bitmap) { val image = FirebaseVisionImage.fromBitmap(bitmap) detector.detectInImage(image) .addOnSuccessListener { result -> result.forEach { label -> val text = label.label val entityId = label.entityId val confidence = label.confidence } } .addOnFailureListener { // Error handling } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Custom TensoreFlow Lite Models Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Determine input and output sizes import tensorflow as tf interpreter = tf.contrib.lite.Interpreter(model_path="my_custom_model.tflite") interpreter.allocate_tensors() # Print input shape and type print(interpreter.get_input_details()[0]['shape']) # Example: [1 224 224 3] print(interpreter.get_input_details()[0]['dtype']) # Example: # Print output shape and type print(interpreter.get_output_details()[0]['shape']) # Example: [1 1000] print(interpreter.get_output_details()[0]['dtype']) # Example: Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

fun configureCustomModel(modelName: String): FirebaseModelInterpreter? { val conditions = FirebaseModelDownloadConditions.Builder() .requireWifi() .build() val cloudSource = FirebaseCloudModelSource.Builder(modelName) .enableModelUpdates(true) .setInitialDownloadConditions(conditions) .setUpdatesDownloadConditions(conditions) .build() FirebaseModelManager.getInstance().registerCloudModelSource(cloudSource) val options = FirebaseModelOptions.Builder() .setCloudModelName(modelName) .setLocalModelName(modelName) .build() return FirebaseModelInterpreter.getInstance(options) } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

fun runInterpreter(input: Any, firebaseInterpreter: FirebaseModelInterpreter) { val inputs = FirebaseModelInputs.Builder() .add(input) // add() as many input arrays as your model requires .build() val inputOutputOptions = FirebaseModelInputOutputOptions.Builder() .setInputFormat(/* input format defined here */) .setOutputFormat(/* output format defined here */) .build() firebaseInterpreter.run(inputs, inputOutputOptions) .addOnSuccessListener { // Interpret results } .addOnFailureListener { // Error handling } } Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018 

Resources → Firebase MLKit - https://firebase.google.com/ docs/ml-kit/ → TensorFlow Lite - https://www.tensorflow.org/lite → TensorFlow models - https://github.com/ tensorflow/models Machine Learning on Mobile with MLKit - @ErikHellman, Øredev 2018