FrenchKit 2017 - Update your reality with Vision an ARKit

Transcript

1. Update your reality with Vision and ARKit Julien Datour
2. Demo time!
3. That’s what I’m talking about Vision ARKit SceneKit
4. Plane detection - ARKit Rectangle detection - Vision Barcode detection - Vision Detecting card size - ARKit
5. Vision
6. –Brett Keating, Apple Manager, during WWDC “High-level on-device solutions to Computer Vision problems through one simple API”
7. –Julien Datour, Me, during FrenchKit “A framework that detects stuff on images”
8. How Vision works
9. The ask VNDetectRectanglesRequest
10. The machinery VNImageRequestHandler
11. The results VNRectangleObservation
12. The ask, the machinery and the results
13. The ask, the machinery and the results let request = VNDetectRectanglesRequest { request, error in guard let observations = request.results as? [VNRectangleObservation] else { return } observations.forEach { observation in let topLeft = observation.topLeft } } let handler = VNImageRequestHandler(ciImage: CIImage(), options: [:]) try? handler.perform([request])
14. The ask, the machinery and the results let request = VNDetectRectanglesRequest { request, error in guard let observations = request.results as? [VNRectangleObservation] else { return } observations.forEach { observation in let topLeft = observation.topLeft } } let handler = VNImageRequestHandler(ciImage: CIImage(), options: [:]) try? handler.perform([request])
15. The ask, the machinery and the results let request = VNDetectRectanglesRequest { request, error in guard let observations = request.results as? [VNRectangleObservation] else { return } observations.forEach { observation in let topLeft = observation.topLeft } } let handler = VNImageRequestHandler(ciImage: CIImage(), options: [:]) try? handler.perform([request])
16. The ask, the machinery and the results let request = VNDetectRectanglesRequest { request, error in guard let observations = request.results as? [VNRectangleObservation] else { return } observations.forEach { observation in let topLeft = observation.topLeft } } let handler = VNImageRequestHandler(ciImage: CIImage(), options: [:]) try? handler.perform([request])
17. The ask, the machinery and the results let request = VNDetectRectanglesRequest { request, error in guard let observations = request.results as? [VNRectangleObservation] else { return } observations.forEach { observation in let topLeft = observation.topLeft } } let handler = VNImageRequestHandler(ciImage: CIImage(), options: [:]) try? handler.perform([request])
18. The ask, the machinery and the barcode let barcodeDetectionRequest = VNDetectBarcodesRequest { request, _ in guard let barcodeObservations = request.results as? [VNBarcodeObservation] else { return } barcodeObservations.forEach { barcodeObservation in guard let payload = barcodeObservation.payloadStringValue else { return } // ... }` } let handler = VNImageRequestHandler(ciImage: CIImage(), options: [:]) try? handler.perform([barcodeDetectionRequest])
19. The ask, the machinery and the barcode let barcodeDetectionRequest = VNDetectBarcodesRequest { request, _ in guard let barcodeObservations = request.results as? [VNBarcodeObservation] else { return } barcodeObservations.forEach { barcodeObservation in guard let payload = barcodeObservation.payloadStringValue else { return } // ... }` } let handler = VNImageRequestHandler(ciImage: CIImage(), options: [:]) try? handler.perform([barcodeDetectionRequest])
21. ARKit - SceneKit
22. SceneKit
23. SCNode A structural representing a position, an orientation and a scale relative to its parent node
24. A plane
25. A sphere
26. Or even text
28. ARKit
29. What ARKit can do for you Scene understanding Plane detection Real-World surface detection
31. Configuring ARKit if ARWorldTrackingConfiguration.isSupported { let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.delegate = self sceneView.session.run(configuration) }
32. Configuring ARKit if ARWorldTrackingConfiguration.isSupported { let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.delegate = self sceneView.session.run(configuration) }
33. Configuring ARKit if ARWorldTrackingConfiguration.isSupported { let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.delegate = self sceneView.session.run(configuration) }
34. Configuring ARKit if ARWorldTrackingConfiguration.isSupported { let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.delegate = self sceneView.session.run(configuration) }
35. Plane detection done right func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let anchor = anchor as? ARPlaneAnchor else { return } let surface = SurfaceNode(anchor: anchor) surfaces[anchor] = surface node.addChildNode(surface) }
38. Hit test let results = sceneView.hitTest(observation.topLeft, types: .existingPlaneUsingExtent)
41. Thanks
42. @JulienDatour