Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Face Recognition with Vision & Core ML

Face Recognition with Vision & Core ML

Simone Civetta

June 20, 2018
Tweet

More Decks by Simone Civetta

Other Decks in Programming

Transcript

  1. Vision ▼ Face and face landmark detection ▼ Text detection

    ▼ Barcode recognition ▼ Feature tracking ▼ Custom Core ML models
  2. Face Rectangle Request let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .right,

    options: [:]) let request = VNDetectFaceRectanglesRequest() try imageRequestHandler.perform([request]) guard let results = request.results as? [VNFaceObservation] else { return } for result in results { result.confidence // 0...1 result.landmarks // Face Landmarks result.boundingBox // Rectangle Around the Face }
  3. Core ML: Execution on CPU or GPU? Diving Deeper ▼

    CPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework
  4. Core ML: Execution on GPU or GPU? Diving Deeper ▼

    GPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework ▽ Abstraction for leveraging the vector-processing capabilities of the CPU
  5. Core ML: Execution on GPU or GPU? Diving Deeper ▼

    GPU or GPU chosen according to the device ▼ CPU execution runs on top of Accelerate framework ▽ Abstraction for leveraging the vector-processing capabilities of the CPU ▼ iOS chooses for You /!\
  6. Core ML Tools Diving Deeper from inception_resnet_v1 import * model

    = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output" ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')
  7. Core ML Tools: Input Normalization and Conversion Diving Deeper from

    inception_resnet_v1 import * model = InceptionResNetV1(weights_path='facenet_keras_weights.h5') coreml_model = coremltools.converters.keras.convert( model, input_names="image", image_input_names="image", output_names="output", add_custom_layers=True, image_scale=2/255.0, red_bias=-1, green_bias=-1, blue_bias=-1, custom_conversion_functions={ "Lambda": convert_lambda } ) coreml_model.save('facenet_keras_weights_coreml.mlmodel')
  8. Core ML in Swift Diving Deeper let model = try

    VNCoreMLModel(for: facenet_keras_weights_coreml().model) let coreMLRequest = VNCoreMLRequest(model: model) // Options coreMLRequest.imageCropAndScaleOption = .scaleFit // ...
  9. Diving Deeper Easier Said Than Done ▼ Input Normalization ▼

    Custom Layers ▼ Input Normalization ▼ Custom Layers ▽ Performance
  10. Custom Layers: No Optimization @objc(scaling) class Scaling: NSObject, MLCustomLayer {

    // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { for i in 0..<inputs.count { let input = inputs[i] let output = outputs[i] assert(input.shape == output.shape) for j in 0..<input.count { let x = input[j].doubleValue let y = x * scale output[j] = NSNumber(value: y) } } } }
  11. Custom Layers: Using Accelerate.framework @objc(scaling) class Scaling: NSObject, MLCustomLayer {

    // ... func evaluate(inputs: [MLMultiArray], outputs: [MLMultiArray]) throws { var scale = Float(self.scale) for i in 0..<inputs.count { let input = inputs[i] let output = outputs[i] assert(input.shape == output.shape) let count = input.count let inputPointer = UnsafeMutablePointer<Float>(OpaquePointer(input.dataPointer)) let outputPointer = UnsafeMutablePointer<Float>(OpaquePointer(output.dataPointer)) vDSP_vsmul(inputPointer, 1, &scale, outputPointer, 1, vDSP_Length(count)) } } }
  12. Custom Layers: Using Metal Shader (GPU) #include <metal_stdlib> using namespace

    metal; kernel void scaling( texture2d_array<half, access::read> inTexture [[texture(0)]], texture2d_array<half, access::write> outTexture [[texture(1)]], constant float& scale [[buffer(0)]], ushort3 gid [[thread_position_in_grid]]) { if (gid.x >= outTexture.get_width() || gid.y >= outTexture.get_height()) { return; } const float4 x = float4(inTexture.read(gid.xy, gid.z)); const float4 y = x * scale; outTexture.write(half4(y), gid.xy, gid.z); }
  13. Custom Layers: Using Metal Shader (GPU) @objc(scaling) class Scaling: NSObject,

    MLCustomLayer { if let encoder = commandBuffer.makeComputeCommandEncoder() { for i in 0..<inputs.count { encoder.setTexture(inputs[i], index: 0) encoder.setTexture(outputs[i], index: 1) var scale = self.scale encoder.setBytes(&scale, length: MemoryLayout<Float>.size, index: 0) encoder.dispatch(pipeline: scalingPipeline, texture: inputs[i]) encoder.endEncoding() } } }
  14. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) {

    let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return }
  15. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) {

    let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return } let pointerNode = SCNNode.createPointerNode(text: personIdentifier) pointerNode.position = point
  16. Adding 3D Objects func paintFaceGeometry(at rect: CGRect, personIdentifier: String) {

    let targetRect = rect.transformed(to: arSceneView.frame.size) let targetRectCenter = CGPoint(x: targetRect.midX, y: targetRect.midY) guard let point = findAverageHitTest(for: targetRectCenter) else { return } let pointerNode = SCNNode.createPointerNode(text: personIdentifier) pointerNode.position = point let constraint = SCNBillboardConstraint() baseNode.constraints = [constraint] arSceneView.scene.rootNode.addChildNode(pointerNode) }
  17. Summing Up What’s Next ▼ Optimizing Model Size ▽ Quantization

    ▼ Better Performances ▼ Core ML 2 (Beta) ▼ Create ML 2.0
  18. Summing Up Recap ▼ What is DL ▼ How to

    use DL for Face Recognition ▼ How to import a face recognition model inside a mobile app ▼ How to make use of a ML model to create an AR experience on a modern phone