Face Recognition with Vision & Core ML

Slide 1

Slide 1 text

Face Recognition on mobile applications Deep Learning & Augmented Reality

Slide 2

Slide 2 text

Integration inside a Mobile App

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

Tell Me Why Diving Deeper Performances Privacy

Slide 6

Slide 6 text

Machine Learning Machine Learning on Mobile

Slide 7

Slide 7 text

Machine Learning Machine Learning on Mobile

Slide 8

Slide 8 text

Machine Learning Machine Learning on Mobile

Slide 9

Slide 9 text

2 Main Solutions 2 Main Solutions

Slide 10

Slide 10 text

Google: TensorFlow 2 Main Solutions

Slide 11

Slide 11 text

Apple: Vision & Core ML 2 Main Solutions

Slide 12

Slide 12 text

Deep Learning for Face Recognition

Slide 13

Slide 13 text

Two Components

Slide 14

Slide 14 text

Machine Learning

Slide 15

Slide 15 text

Vision ▼ Face and face landmark detection ▼ Text detection ▼ Barcode recognition ▼ Feature tracking ▼ Custom Core ML models

Slide 16

Slide 16 text

Face Rectangle Request let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .right, options: [:]) let request = VNDetectFaceRectanglesRequest() try imageRequestHandler.perform([request]) guard let results = request.results as? [VNFaceObservation] else { return } for result in results { result.confidence // 0...1 result.landmarks // Face Landmarks result.boundingBox // Rectangle Around the Face }

Slide 17

Slide 17 text

Machine Learning

Slide 18

Slide 18 text

Apple: Core ML Diving Deeper

Slide 19

Slide 19 text

Core ML: Execution on CPU or GPU? Diving Deeper ▼ CPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework

Slide 20

Slide 20 text

Core ML: Execution on GPU or GPU? Diving Deeper ▼ GPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework ▽ Abstraction for leveraging the vector-processing capabilities of the CPU

Slide 21

Slide 21 text

Slide 22

Slide 22 text

Apple: Core ML Diving Deeper

Slide 23

Slide 23 text

Apple: Core ML Diving Deeper

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

Apple: Core ML Diving Deeper

Slide 26

Slide 26 text

Diving Deeper Main Hurdles ▼ Conversion ▼ Performances ▼ Model Size

Slide 27

Slide 27 text

Apple: Core ML Diving Deeper

Slide 28

Slide 28 text

Core ML Tools Diving Deeper https://github.com/apple/coremltools ▼ Convert ▼ Predict (Test)

Slide 29

Slide 29 text

Core ML Tools Diving Deeper from inception_resnet_v1 import * model = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output" ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')

Slide 30

Slide 30 text

Diving Deeper Easier Said Than Done ▼ Input Normalization ▼ Custom Layers

Slide 31

Slide 31 text

Custom Layers // ... def scaling(x, scale): return x * scale // ...

Slide 32

Slide 32 text

Core ML Tools: Input Normalization and Conversion Diving Deeper from inception_resnet_v1 import * model = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output", add_custom_layers=True, image_scale=2/255.0, red_bias=-1, green_bias=-1, blue_bias=-1, custom_conversion_functions={ "Lambda": convert_lambda } ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')

Slide 33

Slide 33 text

Core ML+ Xcode Diving Deeper

Slide 34

Slide 34 text

Core ML in Swift Diving Deeper let model = try VNCoreMLModel(for: facenet_keras_weights_coreml().model) let coreMLRequest = VNCoreMLRequest(model: model) // Options coreMLRequest.imageCropAndScaleOption = .scaleFit // ...

Slide 35

Slide 35 text

Diving Deeper Easier Said Than Done ▼ Input Normalization ▼ Custom Layers ▼ Input Normalization ▼ Custom Layers ▽ Performance

Slide 36

Slide 36 text

Custom Layers // ... def scaling(x, scale): return x * scale // ...

Slide 37

Slide 37 text

Custom Layers: No Optimization @objc(scaling) class Scaling: NSObject, MLCustomLayer { // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { for i in 0..

Slide 38

Slide 38 text

Custom Layers: No Optimization

Slide 39

Slide 39 text

Custom Layers: Using Accelerate.framework @objc(scaling) class Scaling: NSObject, MLCustomLayer { // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { var scale = Float(self.scale) for i in 0..(OpaquePointer(input.dataPointer)) let outputPointer = UnsafeMutablePointer(OpaquePointer(output.dataPointer)) vDSP_vsmul(inputPointer, 1, &scale, outputPointer, 1, vDSP_Length(count)) } } }

Slide 40

Slide 40 text

Custom Layers: Using Accelerate.framework

Slide 41

Slide 41 text

Custom Layers: Using Metal Shader (GPU) #include using namespace metal; kernel void scaling( texture2d_array inTexture [[texture(0)]], texture2d_array outTexture [[texture(1)]], constant float& scale [[buffer(0)]], ushort3 gid [[thread_position_in_grid]]) { if (gid.x >= outTexture.get_width() || gid.y >= outTexture.get_height()) { return; } const float4 x = float4(inTexture.read(gid.xy, gid.z)); const float4 y = x * scale; outTexture.write(half4(y), gid.xy, gid.z); }

Slide 42

Slide 42 text

Custom Layers: Using Metal Shader (GPU) @objc(scaling) class Scaling: NSObject, MLCustomLayer { if let encoder = commandBuffer.makeComputeCommandEncoder() { for i in 0...size, index: 0) encoder.dispatch(pipeline: scalingPipeline, texture: inputs[i]) encoder.endEncoding() } } }

Slide 43

Slide 43 text

Custom Layers: Using Metal Shader (GPU)

Slide 44

Slide 44 text

Two Components

Slide 45

Slide 45 text

No content

Slide 46

Slide 46 text

Augmented Reality

Slide 47

Slide 47 text

Augmented Reality

Slide 48

Slide 48 text

Augmented Reality

Slide 49

Slide 49 text

Augmented Reality = +

Slide 50

Slide 50 text

Augmented Reality Vision Framework

Slide 51

Slide 51 text

Augmented Reality

Slide 52

Slide 52 text

Adding 3D Objects: What’s a Node? SCNNode()

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) { let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return } let pointerNode = SCNNode.createPointerNode(text: personIdentifier) pointerNode.position = point

Slide 55

Slide 55 text

Slide 56

Slide 56 text

What’s Next

Slide 57

Slide 57 text

Summing Up What’s Next ▼ Optimizing Model Size ▽ Quantization ▼ Better Performances ▼ Core ML 2 (Beta) ▼ Create ML 2.0

Slide 58

Slide 58 text

Summing Up

Slide 59

Slide 59 text

Summing Up Recap ▼ What is DL ▼ How to use DL for Face Recognition ▼ How to import a face recognition model inside a mobile app ▼ How to make use of a ML model to create an AR experience on a modern phone

Slide 60

Slide 60 text

References

Slide 61

Slide 61 text

References References ▼ http://machinethink.net/blog/ ▼ https://developer.apple.com/wwdc/ REF