Face Recognition with Vision & Core ML

Face Recognition with Vision & Core ML

3f309c992e2b1a5c3c014e63810a2f68?s=128

Simone Civetta

June 20, 2018
Tweet

Transcript

  1. Face Recognition on mobile applications Deep Learning & Augmented Reality

  2. Integration inside a Mobile App

  3. None
  4. None
  5. Tell Me Why Diving Deeper Performances Privacy

  6. Machine Learning Machine Learning on Mobile

  7. Machine Learning Machine Learning on Mobile

  8. Machine Learning Machine Learning on Mobile

  9. 2 Main Solutions 2 Main Solutions

  10. Google: TensorFlow 2 Main Solutions

  11. Apple: Vision & Core ML 2 Main Solutions

  12. Deep Learning for Face Recognition

  13. Two Components

  14. Machine Learning

  15. Vision ▼ Face and face landmark detection ▼ Text detection

    ▼ Barcode recognition ▼ Feature tracking ▼ Custom Core ML models
  16. Face Rectangle Request let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .right,

    options: [:]) let request = VNDetectFaceRectanglesRequest() try imageRequestHandler.perform([request]) guard let results = request.results as? [VNFaceObservation] else { return } for result in results { result.confidence // 0...1 result.landmarks // Face Landmarks result.boundingBox // Rectangle Around the Face }
  17. Machine Learning

  18. Apple: Core ML Diving Deeper

  19. Core ML: Execution on CPU or GPU? Diving Deeper ▼

    CPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework
  20. Core ML: Execution on GPU or GPU? Diving Deeper ▼

    GPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework ▽ Abstraction for leveraging the vector-processing capabilities of the CPU
  21. Core ML: Execution on GPU or GPU? Diving Deeper ▼

    GPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework ▽ Abstraction for leveraging the vector-processing capabilities of the CPU ▼ iOS chooses for You /!\
  22. Apple: Core ML Diving Deeper

  23. Apple: Core ML Diving Deeper

  24. None
  25. Apple: Core ML Diving Deeper

  26. Diving Deeper Main Hurdles ▼ Conversion ▼ Performances ▼ Model

    Size
  27. Apple: Core ML Diving Deeper

  28. Core ML Tools Diving Deeper https://github.com/apple/coremltools ▼ Convert ▼ Predict

    (Test)
  29. Core ML Tools Diving Deeper from inception_resnet_v1 import * model

    = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output" ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')
  30. Diving Deeper Easier Said Than Done ▼ Input Normalization ▼

    Custom Layers
  31. Custom Layers // ... def scaling(x, scale): return x *

    scale // ...
  32. Core ML Tools: Input Normalization and Conversion Diving Deeper from

    inception_resnet_v1 import * model = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output", add_custom_layers=True, image_scale=2/255.0, red_bias=-1, green_bias=-1, blue_bias=-1, custom_conversion_functions={ "Lambda": convert_lambda } ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')
  33. Core ML+ Xcode Diving Deeper

  34. Core ML in Swift Diving Deeper let model = try

    VNCoreMLModel(for: facenet_keras_weights_coreml().model) let coreMLRequest = VNCoreMLRequest(model: model) // Options coreMLRequest.imageCropAndScaleOption = .scaleFit // ...
  35. Diving Deeper Easier Said Than Done ▼ Input Normalization ▼

    Custom Layers ▼ Input Normalization ▼ Custom Layers ▽ Performance
  36. Custom Layers // ... def scaling(x, scale): return x *

    scale // ...
  37. Custom Layers: No Optimization @objc(scaling) class Scaling: NSObject, MLCustomLayer {

    // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { for i in 0..<inputs.count { let input = inputs[i] let output = outputs[i] assert(input.shape == output.shape) for j in 0..<input.count { let x = input[j].doubleValue let y = x * scale output[j] = NSNumber(value: y) } } } }
  38. Custom Layers: No Optimization

  39. Custom Layers: Using Accelerate.framework @objc(scaling) class Scaling: NSObject, MLCustomLayer {

    // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { var scale = Float(self.scale) for i in 0..<inputs.count { let input = inputs[i] let output = outputs[i] assert(input.shape == output.shape) let count = input.count let inputPointer = UnsafeMutablePointer<Float>(OpaquePointer(input.dataPointer)) let outputPointer = UnsafeMutablePointer<Float>(OpaquePointer(output.dataPointer)) vDSP_vsmul(inputPointer, 1, &scale, outputPointer, 1, vDSP_Length(count)) } } }
  40. Custom Layers: Using Accelerate.framework

  41. Custom Layers: Using Metal Shader (GPU) #include <metal_stdlib> using namespace

    metal; kernel void scaling( texture2d_array<half, access::read> inTexture [[texture(0)]], texture2d_array<half, access::write> outTexture [[texture(1)]], constant float& scale [[buffer(0)]], ushort3 gid [[thread_position_in_grid]]) { if (gid.x >= outTexture.get_width() || gid.y >= outTexture.get_height()) { return; } const float4 x = float4(inTexture.read(gid.xy, gid.z)); const float4 y = x * scale; outTexture.write(half4(y), gid.xy, gid.z); }
  42. Custom Layers: Using Metal Shader (GPU) @objc(scaling) class Scaling: NSObject,

    MLCustomLayer { if let encoder = commandBuffer.makeComputeCommandEncoder() { for i in 0..<inputs.count { encoder.setTexture(inputs[i], index: 0) encoder.setTexture(outputs[i], index: 1) var scale = self.scale encoder.setBytes(&scale, length: MemoryLayout<Float>.size, index: 0) encoder.dispatch(pipeline: scalingPipeline, texture: inputs[i]) encoder.endEncoding() } } }
  43. Custom Layers: Using Metal Shader (GPU)

  44. Two Components

  45. None
  46. Augmented Reality

  47. Augmented Reality

  48. Augmented Reality

  49. Augmented Reality = +

  50. Augmented Reality Vision Framework

  51. Augmented Reality

  52. Adding 3D Objects: What’s a Node? SCNNode()

  53. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) {

    let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return }
  54. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) {

    let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return } let pointerNode = SCNNode.createPointerNode(text: personIdentifier) pointerNode.position = point
  55. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) {

    let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return } let pointerNode = SCNNode.createPointerNode(text: personIdentifier) pointerNode.position = point let constraint = SCNBillboardConstraint() baseNode.constraints = [constraint] arSceneView.scene.rootNode.addChildNode(pointerNode) }
  56. What’s Next

  57. Summing Up What’s Next ▼ Optimizing Model Size ▽ Quantization

    ▼ Better Performances ▼ Core ML 2 (Beta) ▼ Create ML 2.0
  58. Summing Up

  59. Summing Up Recap ▼ What is DL ▼ How to

    use DL for Face Recognition ▼ How to import a face recognition model inside a mobile app ▼ How to make use of a ML model to create an AR experience on a modern phone
  60. References

  61. References References ▼ http://machinethink.net/blog/ ▼ https://developer.apple.com/wwdc/ REF