Vision for iOS

A brief introduction to Apples computer vision framework with examples of how we can use it get our devices seeing and processing the world around them.

Transcript

1. Andy Bell @andybel
2. What is Vision? #
3. The Vision framework performs face and face landmark detection, text detection, barcode recognition, image registration, and general feature tracking. Vision also allows the use of custom Core ML models for tasks like classification or object detection.
4. Image Classification: Object Recognition: Detecting information in an entire image or video frame Detecting bounding box and classification label for features within an entire image or video frame
10. Detecting Objects in still images #
11. VNImageRequestHandler VNObservation VNRequest - handles the type of recognition required - performs the request(s) on the input data - data for the recognised object(s)
12. What can we detect using VNRequest? Rectangles
13. What can we detect using VNRequest? Faces
14. What can we detect using VNRequest? Faces
15. What can we detect using VNRequest? Faces
16. Yaw Roll Pitch What can we detect using VNRequest? Faces
17. What can we detect using VNRequest? Text + Characters
18. What can we detect using VNRequest? Barcodes .symbologies .payloadString “4037300103328”
19. What can we detect using VNRequest? Barcodes .payloadString .symbologies barcodeDetectRequest.symbologies = [.QR, .Aztec, .UPCE, .Code93FullASCII]
20. What can we detect using VNRequest? Horizon 0.0567 .angle
21. What can we detect using VNRequest? Objects for CoreML classification
22. Things to be aware of
23. CGImage What VNImageRequestHandler accepts as input: CIImage CVPixelBuffer NSURL NSData
24. VNObservation uses normalised coordinate space {0, 0} {1, 1} {0, 1} {1, 0} Normalised coordinate space: (Origin is always lower-left corner)
25. Image Orientation is important EXIF orientation defines which way up the device camera is currently being held
26. When using CoreML, Vision performs all image scaling and colour conversion for you e.g. if your model expects 300 x 300 RGB images
27. let faceRequest = VNDetectFaceRectanglesRequest(completionHandler: handleDetectedFaces) requests.append(faceRequest) var requests = [VNImageBasedRequest]() Some examples
28. let request = VNDetectBarcodesRequest() let roi = CGRect(...) request.regionOfInterest = roi Use regionOfInterest to focus on specific area of input image
29. DispatchQueue.global(qos: .userInitiated).async { do { try imageRequestHandler.perform(requests) } catch { print(error.localizedDescription)) return } } let requests: [VNImageBasedRequest] = [faceRequest, textRequest] let imageRequestHandler = VNImageRequestHandler(cgImage: image, orientation: orientation, options: [:])
30. if let model = try? VNCoreMLModel(for: MobileNet().model) { let coreMlRequest = VNCoreMLRequest(model: model, completionHandler: coreMLhandler) requests.append(coreMlRequest) } else { print("Could not create model for classifier") } let requests = [VNImageBasedRequest]()
31. Still Images Demo…
32. Detecting Objects in image sequences #
33. func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) { guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return } let deviceOrientation = exifOrientationForCurrentDeviceOrientation() // create Vision request with the pixel buffer & orientation… } AVCapture session
34. let requestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: deviceOrientation, options: [:]) do { try requestHandler.perform(requests) } catch { print(error) } AVCapture session
35. Image sequence Demo…
36. Tracking Objects in image sequences #
37. VNImageRequestHandler VNObservation VNRequest
38. VNObservation VNTrackObjectRequest VNSequenceRequestHandler
39. The VNTrackObjectRequest class requires a detected object observation to initialize.
40. Tracking Vs. Re-detection - Tracking has lighter algorithm - Applies temporal smoothing
41. Useful links... #
42. https://developer.apple.com/documentation/vision Main Apple Vision docs
43. WWDC 2018 Session 716 - Object Tracking in Vision Session 717 - Vision with Core ML WWDC 2017 Session 506 - Vision Framework: Building on Core ML
44. Behind the Magic: How we built the ARKit Sudoku Solver https://blog.prototypr.io/behind-the-magic-how- we-built-the-arkit-sudoku-solver-e586e5b685b0
45. Real-time object detection with YOLO http://machinethink.net/blog/object-detection- with-yolo/ YOLO - You Only Look Once https://pjreddie.com/darknet/yolo/
46. https://github.com/likedan/Awesome-CoreML-Models Collection of useful CoreML ready models
47. Andy Bell @andybel Thank you!