Building a "Minimum Viable Product" (MVP) with Face recognition and AR in Android @ Droidcon London 2017

Droidcon London 2017 Talk the 27th of October. Introduction to a use case for building a "Minimum viable product" (MVP), talking about machine learning (face recognition) on Android devices with Augmented reality (AR). You will learn how to plan those kind of projects, the process to follow and future work for those specific projects. Indicating "how", "what" and "why".

Transcript

1. Raúl Hernández López Software Engineer focused on Android Building a “Minimum Viable Product” (MVP) with Face Recognition and AR in Android @RaulHernandezL droidcon London 2017 raulh82vlc raul.h82 #droidconUK
2. A User Story @RaulHernandezL #MVP #droidconUK
3. How? @RaulHernandezL #MVP #droidconUK
4. First things first: Proof of concept @RaulHernandezL #MVP #droidconUK
5. @RaulHernandezL #MVP #droidconUK Face detection Proof of concept
6. @RaulHernandezL #MVP #droidconUK Face detection Augmented Reality Proof of concept
7. @RaulHernandezL #MVP #droidconUK Face recognition Proof of concept
8. @RaulHernandezL #MVP #droidconUK Brainstorm
9. @RaulHernandezL A proper hat with a bow tie Brainstorm #MVP #droidconUK
10. @RaulHernandezL Brainstorm ...also a smoking pipe and why not a smart moustache and a monocle to impress!!! #MVP #droidconUK
11. Use case @RaulHernandezL #MVP #droidconUK Is this what we want???
12. Use case @RaulHernandezL Definitely too complex Stop it! #MVP #droidconUK
13. What do we really need to achieve? @RaulHernandezL #MVP #droidconUK
14. What are the goals? @RaulHernandezL #MVP #droidconUK Build a Prototype
15. What are the goals? @RaulHernandezL #MVP #droidconUK Build a Prototype Something functional
16. Build a Prototype Something functional What are the goals? @RaulHernandezL (but simple!) #MVP #droidconUK
17. What are the goals? @RaulHernandezL Build a Prototype Something functional Easy to be extended #MVP #droidconUK
18. And remember @RaulHernandezL #MVP #droidconUK simple!
19. Use case @RaulHernandezL #AR #MachineLearning #droidconUK Kind of...
20. High level Requirements @RaulHernandezL Selfie camera #AR #MachineLearning #droidconUK
21. High level Requirements @RaulHernandezL #AR #MachineLearning #droidconUK Face detection with Points of interest (POIs) to show AR
22. High level Requirements @RaulHernandezL #AR #MachineLearning #droidconUK Face recognition Raul
23. How do I start now? @RaulHernandezL #AR #MachineLearning #droidconUK
24. How do I start to build my detection system? @RaulHernandezL #AR #MachineLearning #droidconUK Choose a machine learning library:
25. OpenCV workflow @RaulHernandezL #AR #MachineLearning #droidconUK https://github.com/raulh82vlc/Image-Detection-Samples
26. Face detection @RaulHernandezL Detection model (cascade classifier) Haar Cascade vs Local Binary Pattern Histograms (LBPH) #AR #MachineLearning #droidconUK https://github.com/raulh82vlc/Image-Detection-Samples
27. @RaulHernandezL #AR #MachineLearning #droidconUK @NonNull private MatOfRect startDetection() { if (absoluteFaceSize == 0) { int height = matrixGray.rows(); if (Math.round(height * RELATIVE_FACE_SIZE) > 0) { absoluteFaceSize = Math.round(height * RELATIVE_FACE_SIZE); } } MatOfRect faces = new MatOfRect(); if (detectorFace != null) { if (matrixGray.height() > 0) { detectorFace.detectMultiScale(matrixGray, faces, 1.1, 2, 2, new Size(absoluteFaceSize, absoluteFaceSize), new Size()); } } return faces; } Image in gray output opencv.domain.FDInteractorImpl Haar Cascade classifier OpenCV face detection
28. @RaulHernandezL #AR #MachineLearning #droidconUK @NonNull private MatOfRect startDetection() { if (absoluteFaceSize == 0) { int height = matrixGray.rows(); if (Math.round(height * RELATIVE_FACE_SIZE) > 0) { absoluteFaceSize = Math.round(height * RELATIVE_FACE_SIZE); } } MatOfRect faces = new MatOfRect(); if (detectorFace != null) { if (matrixGray.height() > 0) { detectorFace.detectMultiScale(matrixGray, faces, 1.1, 2, 2, new Size(absoluteFaceSize, absoluteFaceSize), new Size()); } } return faces; } opencv.domain.FDInteractorImpl OpenCV face detection
29. Rendering graphics @RaulHernandezL Computer vision general purpose library #AR #MachineLearning #droidconUK
30. @RaulHernandezL #AR #MachineLearning #droidconUK // computing eye areas as well as splitting it Rect leftEyeArea = new Rect(face.x + face.width / 16 + (face.width - 2 * face.width / 16) / 2, (int) (face.y + (face.height / 4.5)), (face.width - 2 * face.width / 16) / 2, (int) (face.height / 3.0)); FaceDrawerOpenCV.drawEyesRectangles(rightEyeArea, leftEyeArea, matrixRGBA); opencv.domain.EyesDetectionInteractorImpl OpenCV draws initial eyes areas
31. @RaulHernandezL #AR #MachineLearning #droidconUK // computing eye areas as well as splitting it Rect leftEyeArea = new Rect(face.x + face.width / 16 + (face.width - 2 * face.width / 16) / 2, (int) (face.y + (face.height / 4.5)), (face.width - 2 * face.width / 16) / 2, (int) (face.height / 3.0)); FaceDrawerOpenCV.drawEyesRectangles(rightEyeArea, leftEyeArea, matrixRGBA); opencv.domain.EyesDetectionInteractorImpl public static void drawIrisRectangle (Rect eyeTemplate , Mat matrixRgba) { Imgproc.rectangle(matrixRgba, eyeTemplate.tl() , eyeTemplate.br() , new Scalar(255, 0, 0, 255), 2); } Image in color OpenCV draws initial eyes areas
32. OpenCV face validation @RaulHernandezL Face (& eyes) detection AR markers #AR #MachineLearning #droidconUK
33. @RaulHernandezL #AR #MachineLearning #droidconUK String methodForEyes; if (learnFrames < LEARN_FRAMES_LIMIT) { templateRight = buildTemplate(rightEyeArea, IRIS_MIN_SIZE, matrixGray, matrixRGBA, detectorEye); templateLeft = buildTemplate(leftEyeArea, IRIS_MIN_SIZE, matrixGray, matrixRGBA, detectorEye); learnFrames++; methodForEyes = "building Template with Detect multiscale, frame: " + learnFrames; } else { // Learning finished, use the new templates for template matching matchEye(rightEyeArea, templateRight, matrixGray, matrixRGBA); matchEye(leftEyeArea, templateLeft, matrixGray, matrixRGBA); methodForEyes = "match eye with Template, frame: " + learnFrames; } notifyEyesFound(methodForEyes); OpenCV eyes detection opencv.domain.EyesDetectionInteractorImpl
34. @RaulHernandezL #AR #MachineLearning #droidconUK String methodForEyes; if (learnFrames < LEARN_FRAMES_LIMIT) { templateRight = buildTemplate(rightEyeArea, IRIS_MIN_SIZE, matrixGray, matrixRGBA, detectorEye); templateLeft = buildTemplate(leftEyeArea, IRIS_MIN_SIZE, matrixGray, matrixRGBA, detectorEye); learnFrames++; methodForEyes = "building Template with Detect multiscale, frame: " + learnFrames; } else { // Learning finished, use the new templates for template matching matchEye(rightEyeArea, templateRight, matrixGray, matrixRGBA); matchEye(leftEyeArea, templateLeft, matrixGray, matrixRGBA); methodForEyes = "match eye with Template, frame: " + learnFrames; } notifyEyesFound(methodForEyes); OpenCV eyes detection opencv.domain.EyesDetectionInteractorImpl
35. @RaulHernandezL #AR #MachineLearning #droidconUK String methodForEyes; if (learnFrames < LEARN_FRAMES_LIMIT) { templateRight = buildTemplate(rightEyeArea, IRIS_MIN_SIZE, matrixGray, matrixRGBA, detectorEye); templateLeft = buildTemplate(leftEyeArea, IRIS_MIN_SIZE, matrixGray, matrixRGBA, detectorEye); learnFrames++; methodForEyes = "building Template with Detect multiscale, frame: " + learnFrames; } else { // Learning finished, use the new templates for template matching matchEye(rightEyeArea, templateRight, matrixGray, matrixRGBA); matchEye(leftEyeArea, templateLeft, matrixGray, matrixRGBA); methodForEyes = "match eye with Template, frame: " + learnFrames; } notifyEyesFound(methodForEyes); OpenCV eyes detection opencv.domain.EyesDetectionInteractorImpl
36. @RaulHernandezL #AR #MachineLearning #droidconUK Imgproc.matchTemplate(submatGray, builtTemplate, outputTemplateMat, Imgproc.TM_SQDIFF_NORMED); Core.MinMaxLocResult minMaxLocResult = Core.minMaxLoc(outputTemplateMat); // when is difference in matching methods, the best match is max / min value matchLoc = minMaxLocResult.minLoc; Point matchLocTx = new Point(matchLoc.x + area.x, matchLoc.y + area.y); Point matchLocTy = new Point(matchLoc.x + builtTemplate.cols() + area.x, matchLoc.y + builtTemplate.rows() + area.y); FaceDrawerOpenCV.drawMatchedEye(matchLocTx, matchLocTy, matrixRGBA); opencv.domain.EyesDetectionInteractorImpl OpenCV template matching
37. @RaulHernandezL #AR #MachineLearning #droidconUK Imgproc.matchTemplate(submatGray, builtTemplate, outputTemplateMat, Imgproc.TM_SQDIFF_NORMED); Core.MinMaxLocResult minMaxLocResult = Core.minMaxLoc(outputTemplateMat); // when is difference in matching methods, the best match is max / min value matchLoc = minMaxLocResult.minLoc; Point matchLocTx = new Point(matchLoc.x + area.x, matchLoc.y + area.y); Point matchLocTy = new Point(matchLoc.x + builtTemplate.cols() + area.x, matchLoc.y + builtTemplate.rows() + area.y); FaceDrawerOpenCV.drawMatchedEye(matchLocTx, matchLocTy, matrixRGBA); opencv.domain.EyesDetectionInteractorImpl OpenCV minMaxLoc
38. @RaulHernandezL #AR #MachineLearning #droidconUK Imgproc.matchTemplate(submatGray, builtTemplate, outputTemplateMat, Imgproc.TM_SQDIFF_NORMED); Core.MinMaxLocResult minMaxLocResult = Core.minMaxLoc(outputTemplateMat); // when is difference in matching methods, the best match is max / min value matchLoc = minMaxLocResult.minLoc; Point matchLocTx = new Point(matchLoc.x + area.x, matchLoc.y + area.y); Point matchLocTy = new Point(matchLoc.x + builtTemplate.cols() + area.x, matchLoc.y + builtTemplate.rows() + area.y); FaceDrawerOpenCV.drawMatchedEye(matchLocTx, matchLocTy, matrixRGBA); opencv.render.FaceDrawerOpenCV OpenCV draw eye public static void drawMatchedEye(Point matchLocTx, Point matchLocTy, Mat matrixRgba) { Imgproc.rectangle(matrixRgba, matchLocTx, matchLocTy, new Scalar(255, 255, 0, 255)); }
39. OpenCV template matching @RaulHernandezL #AR #MachineLearning #droidconUK cv2.TM_SQDIFF Original picture from: opencv 3.0 Documentation
40. OpenCV face validation @RaulHernandezL Accurate Eye detection AR markers #AR #MachineLearning #droidconUK
41. @RaulHernandezL OpenCV UX #AR #MachineLearning #droidconUK
42. How do I start? @RaulHernandezL Native camera detection Camera API Camera 2 API #AR #MachineLearning #droidconUK https://github.com/raulh82vlc/Image-Detection-Samples
43. Camera2 workflow @RaulHernandezL #AR #MachineLearning #droidconUK https://github.com/raulh82vlc/Image-Detection-Samples
44. Rendering graphics @RaulHernandezL Canvas rendering #AR #MachineLearning #droidconUK
45. Rendering graphics @RaulHernandezL OpenGL or external Graphics libraries #AR #MachineLearning #droidconUK
46. private CameraCaptureSession.CaptureCallback captureCallback = new CameraCaptureSession.CaptureCallback() { … @Override public void onCaptureProgressed(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull CaptureResult captureResult) { detectFaces(captureResult); } @Override public void onCaptureCompleted(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull TotalCaptureResult captureResult) { detectFaces(captureResult); } @RaulHernandezL #AR #MachineLearning #droidconUK Camera2 face detection camera2.presentation.FDCamera2Presenter
47. private CameraCaptureSession.CaptureCallback captureCallback = new CameraCaptureSession.CaptureCallback() { … @Override public void onCaptureProgressed(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull CaptureResult captureResult) { detectFaces(captureResult); } @Override public void onCaptureCompleted(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull TotalCaptureResult captureResult) { detectFaces(captureResult); } @RaulHernandezL #AR #MachineLearning #droidconUK Camera2 face detection camera2.presentation.FDCamera2Presenter
48. private CameraCaptureSession.CaptureCallback captureCallback = new CameraCaptureSession.CaptureCallback() { … @Override public void onCaptureProgressed(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull CaptureResult captureResult) { detectFaces(captureResult); } @Override public void onCaptureCompleted(@NonNull CameraCaptureSession session, @NonNull CaptureRequest request, @NonNull TotalCaptureResult captureResult) { detectFaces(captureResult); } @RaulHernandezL #AR #MachineLearning #droidconUK Camera2 face detection camera2.presentation.FDCamera2Presenter
49. private void detectFaces(CaptureResult captureResult) { Integer mode = captureResult.get(CaptureResult.STATISTICS_FACE_DETECT_MODE); if (isViewAvailable() && mode != null) { android.hardware.camera2.params.Face[] faces = captureResult.get(CaptureResult.STATISTICS_FACES); if (faces != null) { Log.i(TAG, "faces : " + faces.length + " , mode : " + mode); for (android.hardware.camera2.params.Face face : faces) { Rect faceBounds = face.getBounds(); // Once processed, the result is sent back to the View presenterView.onFaceDetected(mapCameraFaceToCanvas(faceBounds, face.getLeftEyePosition(), face.getRightEyePosition())); } } } } @RaulHernandezL #AR #MachineLearning #droidconUK Camera2 face detection camera2.presentation.FDCamera2Presenter
50. private void detectFaces(CaptureResult captureResult) { Integer mode = captureResult.get(CaptureResult.STATISTICS_FACE_DETECT_MODE); if (isViewAvailable() && mode != null) { android.hardware.camera2.params.Face[] faces = captureResult.get(CaptureResult.STATISTICS_FACES); if (faces != null) { Log.i(TAG, "faces : " + faces.length + " , mode : " + mode); for (android.hardware.camera2.params.Face face : faces) { Rect faceBounds = face.getBounds(); // Once processed, the result is sent back to the View presenterView.onFaceDetected(mapCameraFaceToCanvas(faceBounds, face.getLeftEyePosition(), face.getRightEyePosition())); } } } } @RaulHernandezL #AR #MachineLearning #droidconUK Camera2 face detection camera2.presentation.FDCamera2Presenter
51. private void detectFaces(CaptureResult captureResult) { Integer mode = captureResult.get(CaptureResult.STATISTICS_FACE_DETECT_MODE); if (isViewAvailable() && mode != null) { android.hardware.camera2.params.Face[] faces = captureResult.get(CaptureResult.STATISTICS_FACES); if (faces != null) { Log.i(TAG, "faces : " + faces.length + " , mode : " + mode); for (android.hardware.camera2.params.Face face : faces) { Rect faceBounds = face.getBounds(); // Once processed, the result is sent back to the View presenterView.onFaceDetected(mapCameraFaceToCanvas(faceBounds, face.getLeftEyePosition(), face.getRightEyePosition())); } } } } @RaulHernandezL #AR #MachineLearning #droidconUK Camera2 face detection camera2.presentation.FDCamera2Presenter
52. private void detectFaces(CaptureResult captureResult) { Integer mode = captureResult.get(CaptureResult.STATISTICS_FACE_DETECT_MODE); if (isViewAvailable() && mode != null) { android.hardware.camera2.params.Face[] faces = captureResult.get(CaptureResult.STATISTICS_FACES); if (faces != null) { Log.i(TAG, "faces : " + faces.length + " , mode : " + mode); for (android.hardware.camera2.params.Face face : faces) { Rect faceBounds = face.getBounds(); // Once processed, the result is sent back to the View presenterView.onFaceDetected(mapCameraFaceToCanvas(faceBounds, face.getLeftEyePosition(), face.getRightEyePosition())); } } } } @RaulHernandezL #AR #MachineLearning #droidconUK Camera2 face detection camera2.presentation.FDCamera2Presenter
53. public class FaceDrawer extends View { ... @Override protected void onDraw(Canvas canvas) { super.onDraw(canvas); if (face != null) { Square faceShape = face.getFaceShape(); int w = faceShape.getWidth(); int h = faceShape.getHeight(); drawFaceMarker(canvas, faceShape, w, h); // Bitmap AR Super Sayan drawBitmapAR(canvas, drawableStore.get(KEY_BITMAP_HEAD), w, h, (int) faceShape.getStart().getxAxis() - (w / 2), (int) faceShape.getStart().getyAxis() - (int)(h * 1.5)); } } @RaulHernandezL #AR #MachineLearning #droidconUK camera2.render.FaceDrawer Camera2 face rendering
54. public class FaceDrawer extends View { ... @Override protected void onDraw(Canvas canvas) { super.onDraw(canvas); if (face != null) { Square faceShape = face.getFaceShape(); int w = faceShape.getWidth(); int h = faceShape.getHeight(); drawFaceMarker(canvas, faceShape, w, h); // Bitmap AR Super Sayan drawBitmapAR(canvas, drawableStore.get(KEY_BITMAP_HEAD), w, h, (int) faceShape.getStart().getxAxis() - (w / 2), (int) faceShape.getStart().getyAxis() - (int)(h * 1.5)); } } @RaulHernandezL #AR #MachineLearning #droidconUK camera2.render.FaceDrawer Camera2 face rendering
55. private void drawBitmapAR(Canvas canvas, Drawable drawable, int w, int h, int x, int y) { if (drawable != null) { int widthGraphic = drawable.getIntrinsicWidth(); int heightGraphic = drawable.getIntrinsicHeight(); setTransformation(w, h, x, y, widthGraphic, heightGraphic); canvas.setMatrix(transformation); drawable.draw(canvas); } } @RaulHernandezL #AR #MachineLearning #droidconUK camera2.render.FaceDrawer Camera2 face rendering
56. private void drawBitmapAR(Canvas canvas, Drawable drawable, int w, int h, int x, int y) { if (drawable != null) { int widthGraphic = drawable.getIntrinsicWidth(); int heightGraphic = drawable.getIntrinsicHeight(); setTransformation(w, h, x, y, widthGraphic, heightGraphic); canvas.setMatrix(transformation); drawable.draw(canvas); } } private void setTransformation(int w, int h, int x, int y, int widthGraphic, int heightGraphic) { MeasuresUI measures = TransformationsHelper.calcMeasures(w, h, viewWidth, viewHeight, x, y, widthGraphic, heightGraphic); transformation.setScale(measures.getScale(), measures.getScale()); transformation.postTranslate(measures.getDx(), measures.getDy()); transformation.postConcat(transformation); } @RaulHernandezL #AR #MachineLearning #droidconUK camera2.render.FaceDrawer Camera2 face rendering
57. @RaulHernandezL Face detection (Camera2) Camera2 face validation #AR #MachineLearning #droidconUK
58. @RaulHernandezL AR marker (Paint) with Canvas on a View #AR #MachineLearning #droidconUK Camera2 face validation
59. Camera2 UX @RaulHernandezL #AR #MachineLearning #droidconUK
60. Can you see any commonalities? @RaulHernandezL #AR #MachineLearning #droidconUK
61. Can you see any commonalities? @RaulHernandezL #AR #MachineLearning #droidconUK Clear and separated features
62. Can you see any commonalities? @RaulHernandezL #AR #MachineLearning #droidconUK Clear and separated features A defined Architecture
63. Can you see any commonalities? @RaulHernandezL #AR #MachineLearning #droidconUK Clear and separated features A defined Architecture UI driven
64. Did we finish this MVP? @RaulHernandezL #AR #MachineLearning #droidconUK
65. @RaulHernandezL #AR #MachineLearning #droidconUK Detecting nose or mouth, or other shapes to render. Improvement of graphics user experience (UX) for instance using a kalman filter, or using tracking algorithms like “good features to track”... Extensions for detection & rendering
66. Is anything missed? @RaulHernandezL #AR #MachineLearning #droidconUK Start planning Face recognition
67. Recognition workflow @RaulHernandezL #AR #MachineLearning #droidconUK
68. Recognition workflow @RaulHernandezL #AR #MachineLearning #droidconUK
69. Machine learning @RaulHernandezL #AR #MachineLearning #droidconUK Microsoft Azure Machine Learning cheat sheet
70. Deep learning @RaulHernandezL #AR #MachineLearning #droidconUK Adapted from the original image from Max Tegmark - MIT Connections between physics and deep learning on YouTube Label Image
71. Convolutional Neural Network (CNN) @RaulHernandezL #AR #MachineLearning #droidconUK Original image from Mathworks Discovery section
72. Recognition Cloud APIs @RaulHernandezL #AR #MachineLearning #droidconUK a. Google Cloud Vision (Only Detection) b. Amazon Rekognition c. Microsoft Azure Cognitive Services Azure Cognitive services for face API Cloud Google Vision API Amazon Rekognition API
73. Train a model from scratch @RaulHernandezL #AR #MachineLearning #droidconUK
74. Train a model from scratch @RaulHernandezL #AR #MachineLearning #droidconUK
75. Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK
76. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK Transfer learning Pic 1 from Tate museum Pic 2 from Mirror Celebrity news Pic 3 from Eccles photographs blog
77. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK Daj´s Using a pre-trained Tensorflow model on Android Original diagram from Jalammar blog -> Supercharging android apps using Tensorflow
78. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK TensorFlow now bundles the native binaries and Java interface.
79. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK TensorFlow now bundles the native binaries and Java interface. JAR -> libandroid_tensorflow_interface_java.jar or native binaries -> libtensorflow_inference.so dependencies { compile 'org.tensorflow:tensorflow-android:1.2.0' }
80. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK Adding to app/src/main/assets: Model graph -> .pb & graph labels -> .txt
81. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK public static TensorFlowFaceClassifier buildFaceClassifier(AssetManager assetManager, String modelFileName, String labelFileName, int inputSize, String inputName, String outputName) { TensorFlowFaceClassifier faceClassifier = new TensorFlowFaceClassifier(); … // Set input and output names // For each line read from labelFileName, add its label into memory faceClassifier.addLabel(lineOnFile); // String values … faceClassifier.inferenceInterface = new TensorFlowInferenceInterface(assetManager, modelFileName); // The shape of the output is [N, NUM_CLASSES], where N is the batch size. int numClasses = (int) faceClassifier.inferenceInterface.graph().operation(outputName).output(0).shape().size(1); faceClassifier.inputSize = inputSize; faceClassifier.outputNames = new String[]{outputName}; faceClassifier.outputs = new float[numClasses]; return faceClassifier; } .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
82. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK public static TensorFlowFaceClassifier buildFaceClassifier(AssetManager assetManager, String modelFileName, String labelFileName, int inputSize, String inputName, String outputName) { TensorFlowFaceClassifier faceClassifier = new TensorFlowFaceClassifier(); … // Set input and output names // For each line read from labelFileName, add its label into memory faceClassifier.addLabel(lineOnFile); // String values … faceClassifier.inferenceInterface = new TensorFlowInferenceInterface(assetManager, modelFileName); // The shape of the output is [N, NUM_CLASSES], where N is the batch size. int numClasses = (int) faceClassifier.inferenceInterface.graph().operation(outputName).output(0).shape().size(1); faceClassifier.inputSize = inputSize; faceClassifier.outputNames = new String[]{outputName}; faceClassifier.outputs = new float[numClasses]; return faceClassifier; } .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
83. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK public static TensorFlowFaceClassifier buildFaceClassifier(AssetManager assetManager, String modelFileName, String labelFileName, int inputSize, String inputName, String outputName) { TensorFlowFaceClassifier faceClassifier = new TensorFlowFaceClassifier(); … // Set input and output names // For each line read from labelFileName, add its label into memory faceClassifier.addLabel(lineOnFile); // String values … faceClassifier.inferenceInterface = new TensorFlowInferenceInterface(assetManager, modelFileName); // The shape of the output is [N, NUM_CLASSES], where N is the batch size. int numClasses = (int) faceClassifier.inferenceInterface.graph().operation(outputName).output(0).shape().size(1); faceClassifier.inputSize = inputSize; faceClassifier.outputNames = new String[]{outputName}; faceClassifier.outputs = new float[numClasses]; return faceClassifier; } Output layer size .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
84. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK public static TensorFlowFaceClassifier buildFaceClassifier(AssetManager assetManager, String modelFileName, String labelFileName, int inputSize, String inputName, String outputName) { TensorFlowFaceClassifier faceClassifier = new TensorFlowFaceClassifier(); … // Set input and output names // For each line read from labelFileName, add its label into memory faceClassifier.addLabel(lineOnFile); // String values … faceClassifier.inferenceInterface = new TensorFlowInferenceInterface(assetManager, modelFileName); // The shape of the output is [N, NUM_CLASSES], where N is the batch size. int numClasses = (int) faceClassifier.inferenceInterface.graph().operation(outputName).output(0).shape().size(1); faceClassifier.inputSize = inputSize; faceClassifier.outputNames = new String[]{outputName}; faceClassifier.outputs = new float[numClasses]; return faceClassifier; } Confidence results .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
85. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK public static TensorFlowFaceClassifier buildFaceClassifier(AssetManager assetManager, String modelFileName, String labelFileName, int inputSize, String inputName, String outputName) { TensorFlowFaceClassifier faceClassifier = new TensorFlowFaceClassifier(); … // Set input and output names // For each line read from labelFileName add its label into memory faceClassifier.addLabel(lineOnFile); // String values … faceClassifier.inferenceInterface = new TensorFlowInferenceInterface(assetManager, modelFileName); // The shape of the output is [N, NUM_CLASSES], where N is the batch size. int numClasses = (int) faceClassifier.inferenceInterface.graph().operation(outputName).output(0).shape().size(1); faceClassifier.inputSize = inputSize; faceClassifier.outputNames = new String[]{outputName}; faceClassifier.outputs = new float[numClasses]; return faceClassifier; } .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
86. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified) public List<Recognition> recognizeFaceFromImage(final float[] pixels) { // Copy the input data into TensorFlow inferenceInterface.feed(inputName, pixels, new long[]{inputSize * inputSize}); // Run the inference call inferenceInterface.run(outputNames, runStatus); // Copy the output Tensor back into the output array inferenceInterface.fetch(outputName, outputs); // Find the best classifications for (int i = 0; i < outputs.length; ++i) { if (outputs[i] > THRESHOLD) { // > 0.1f // the following priority queue is sorted by resulting confidence priorityQueue.add(new Recognition( "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null)); } } return getRecognitionsListFromQueue(priorityQueue); } .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
87. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified) public List<Recognition> recognizeFaceFromImage(final float[] pixels) { // Copy the input data into TensorFlow inferenceInterface.feed(inputName, pixels, new long[]{inputSize * inputSize}); // Run the inference call inferenceInterface.run(outputNames, runStatus); // Copy the output Tensor back into the output array inferenceInterface.fetch(outputName, outputs); // Find the best classifications for (int i = 0; i < outputs.length; ++i) { if (outputs[i] > THRESHOLD) { // > 0.1f // the following priority queue is sorted by resulting confidence priorityQueue.add(new Recognition( "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null)); } } return getRecognitionsListFromQueue(priorityQueue); } .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
88. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified) public List<Recognition> recognizeFaceFromImage(final float[] pixels) { // Copy the input data into TensorFlow inferenceInterface.feed(inputName, pixels, new long[]{inputSize * inputSize}); // Run the inference call inferenceInterface.run(outputNames, runStatus); // Copy the output Tensor back into the output array inferenceInterface.fetch(outputName, outputs); // Find the best classifications for (int i = 0; i < outputs.length; ++i) { if (outputs[i] > THRESHOLD) { // > 0.1f // the following priority queue is sorted by resulting confidence priorityQueue.add(new Recognition( "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i])); } } return getRecognitionsListFromQueue(priorityQueue); } Extracted values of confidence .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
89. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified) public List<Recognition> recognizeFaceFromImage(final float[] pixels) { // Copy the input data into TensorFlow inferenceInterface.feed(inputName, pixels, new long[]{inputSize * inputSize}); // Run the inference call inferenceInterface.run(outputNames, runStatus); // Copy the output Tensor back into the output array inferenceInterface.fetch(outputName, outputs); // Find the best classifications for (int i = 0; i < outputs.length; ++i) { if (outputs[i] > THRESHOLD) { // > 0.1f // the following priority queue is sorted by resulting confidence priorityQueue.add(new Recognition( "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i])); } } return getRecognitionsListFromQueue(priorityQueue); } Id Label associated to this recognition Confidence .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
90. Using a Pre-trained model @RaulHernandezL #AR #MachineLearning #droidconUK .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified) public List<Recognition> recognizeFaceFromImage(final float[] pixels) { // Copy the input data into TensorFlow inferenceInterface.feed(inputName, pixels, new long[]{inputSize * inputSize}); // Run the inference call inferenceInterface.run(outputNames, runStatus); // Copy the output Tensor back into the output array inferenceInterface.fetch(outputName, outputs); // Find the best classifications for (int i = 0; i < outputs.length; ++i) { if (outputs[i] > THRESHOLD) { // > 0.1f // the following priority queue is sorted by resulting confidence priorityQueue.add(new Recognition( "" + i, labels.size() > i ? labels.get(i) : "unknown", outputs[i], null)); } } return getRecognitionsListFromQueue(priorityQueue); } .facerecognition.TensorFlowFaceClassifier (AndroidTensorFlowMNISTExample modified)
91. Output @RaulHernandezL #AR #MachineLearning #droidconUK
92. Face Recognition @RaulHernandezL #AR #MachineLearning #droidconUK Expected status: Face is recognized Use case image taken when using: Qualeams Android Face Recognition with Deep Learning Test Framework
93. Conclusions @RaulHernandezL #MVP #droidconUK
94. Conclusions @RaulHernandezL #MVP #droidconUK Make something simple (a prototype) means take right decisions on time
95. Conclusions @RaulHernandezL #MVP #droidconUK Make something functional means we have enough insights, time and learnt lessons to build something ready to go
96. Conclusions @RaulHernandezL #MVP #droidconUK Make something extendable means we have made an effort to introduce best practices to create a maintainable product
97. References @RaulHernandezL Image Detection & AR Presentation Samples: https://github.com/raulh82vlc/Image-Detection-Samples OpenCV OpenCV tutorial http://www.learnopencv.com/image-recognition-and-object-detection-part1/ OpenCV Sample https://github.com/opencv/opencv/tree/master/samples/android/face-detection OpenCV for Secret Agents - Joseph Howse https://www.packtpub.com/application-development/opencv-secret-agents Camera 2 Camera 2 Sample Basic https://github.com/googlesamples/android-Camera2Basic Camera 2 Reference https://developer.android.com/reference/android/hardware/camera2/package-summary.html Camera 2 Introduction on Dev.Bytes https://youtu.be/Xtp3tH27OFs #AR #MachineLearning #droidconUK
98. References @RaulHernandezL TensorFlow Deep learning library https://github.com/Qualeams/Android-Face-Recognition-with-Deep-Learning-Library Mindorks MNIST example https://github.com/MindorksOpenSource/AndroidTensorFlowMNISTExample ML LetterPredictor Android https://github.com/mccorby/ML-LetterPredictor-Android TensorFlow image retraining https://www.tensorflow.org/tutorials/image_retraining Applied TensorFlow for Android apps https://speakerdeck.com/daj/applied-tensorflow-for-android-apps Building Mobile Apps with TensorFlow - Pete Warden Machine learning Deep Learning in a Nutshell – what it is, how it works, why care? https://www.kdnuggets.com/2015/01/deep-learning-explanation-what-how-why.html Transfer learning with Keras https://medium.com/towards-data-science/transfer-learning-using-keras-d804b2e04ef8 Neural networks deep learning https://medium.com/machine-learning-for-humans/neural-networks-deep-learning-cdad8aeae49b #AR #MachineLearning #droidconUK
99. Thanks! QA #droidconUK @RaulHernandezL raulh82vlc raul.h82