Enriched UI using ARKit

Slides of my talk at SwiftConf 2017 in Cologne. Video available at http://bit.ly/2eRnUvF

Transcript

1. @rompelstilchen ENRICHED UI WITH ARKIT HUGUES BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 1
2. @rompelstilchen ENRICHED UI WITH ARKIT TOPICS ARKit Overview / Primer SpriteKit / SceneKit Primer Sample UI Demo Limitations Going further 2
3. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT OVERVIEW Visual Inertial Odometry Inertial Measurement Unit Dead reckoning 6D (3D translation/3D rotation) 3
4. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT OVERVIEW 2D Plane detection Metric scale Light estimation Integration with SpriteKit, SceneKit, Metal or custom 4
5. @rompelstilchen 5
6. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER ARSessionConfiguration ARSession ARFrame ARAnchor 6
7. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARSESSION Manage AR processing Reset Tracking Session Updates 7
8. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARFRAME Camera image to render the background of the scene Tracking informations to find device's location, orientation and state Scene informations such as light estimate and location in space 8
9. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARANCHOR Real-world position and orientation Managed via ARSession Also created for you by ARKit 9
10. @rompelstilchen SPRITEKIT VS SCENEKIT 10
11. @rompelstilchen ENRICHED UI WITH SPRITEKIT HUGUES BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 11
12. @rompelstilchen ENRICHED UI WITH ARKIT SPRITEKIT - PRIMER SKView SKScene SKNode (SKSprite, SKTextNode, …) (SKAction, SKTransition, SKTexture, ..) 12
13. @rompelstilchen ENRICHED UI WITH ARKIT SPRITEKIT / ARKIT ARSKView > SKView + ARSession ARAnchor <> SKNode That’s pretty much it! 13
14. @rompelstilchen DEMO ENRICHED UI WITH ARKIT / SPRITEKIT 14
15. @rompelstilchen ENRICHED UI WITH ARKIT SPRITEKIT / ARKIT - STEPS Position content using ARAnchor Interact with content using UIGesture, UIResponder or camera position tracking Convert point back and forth ARKit do the rest 15
16. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - FIX SCREEN POSITION // Fix content in the center of the screen let node = SKShapeNode(rectOf: CGSize(width: 20, height: 20), cornerRadius: 10) node.fillColor = .blue sceneView.scene?.addChild(node) // it’s a UIView subclass let label = UILabel() sceneView.addSubview(label) 16 @rompelstilchen
17. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO CAMERA CURRENT POSITION if let camera = session.currentFrame?.camera { var translation = matrix_identity_float4x4 translation.columns.3.z = -1 let transform = simd_mul(camera.transform, translation) let anchor = ARAnchor(transform: transform) session.add(anchor: anchor) } func view(_ view: ARSKView, nodeFor anchor: ARAnchor) -> SKNode? { return SKLabelNode(text: “Hello World") } 17 @rompelstilchen
18. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO REAL WORLD (PLANE) 18 @rompelstilchen // on touch let location = gesture.location(in: sceneView) // plane detection enabled and detected let results = sceneView.hitTest(location, types: .existingPlane) if let anchor = results.first?.anchor { sceneView.session.add(anchor: anchor) } func view(_ view: ARSKView, nodeFor anchor: ARAnchor) -> SKNode? { return SKLabelNode(text: “Hello World") }
19. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - UIKIT/SPRITEKIT class SKTouch: SKShapeNode { override init() { super.init() isUserInteractionEnabled = true } override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) { … } } 19 @rompelstilchen
20. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - UIKIT/SPRITEKIT 20 @rompelstilchen @objc func tap_(gesture: UITapGestureRecognizer) { guard gesture.state == .ended else { return } let location = gesture.location(in: sceneView) let nodes = sceneView.scene?.nodes(at: location) if let node = nodes?.first { … } }
21. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - DEVICE 21 @rompelstilchen func session(_ session: ARSession, didUpdate frame: ARFrame) { let location = sceneView.center if let sceneLocation = sceneView.scene?.convertPoint(fromView: location), let nodes = sceneView.scene?.nodes(at: sceneLocation) { if let node = nodes.first { ... } } }
22. @rompelstilchen ENRICHED UI WITH SCENEKIT HUGUES BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 22
23. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT - PRIMER SCNView SCNScene SCNNode (SCNSprite, SCNTextNode, …) SCNGeometry, SCNMaterial 23
24. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT / ARKIT ARSCNView > SCNView + ARSession ARAnchor <> SCNNode That’s pretty much it (again) 24
25. @rompelstilchen DEMO ENRICHED UI WITH ARKIT / SCENEKIT 25
26. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT / ARKIT - STEPS Position content using ARAnchor and/or the rootNode (initial camera position) Interact with content using UIGesture or the device FOV/position Convert point back and forth ARKit do the rest 26
27. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - FIX POSITION // fix position relative to the point of view // kind of useless let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0) box.firstMaterial?.diffuse.contents = UIColor.red let node = SCNNode(geometry: box) node.position = SCNVector3(0, 0, -1) sceneView.pointOfView?.addChildNode( node) // it’s a UIView subclass let label = UILabel() view.addSubview(label) 27 @rompelstilchen
28. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO INITIAL CAMERA POSITION 28 @rompelstilchen // the root node is updated relative to camera by ARKit let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0) box.firstMaterial?.diffuse.contents = UIColor.red let node = SCNNode(geometry: box) node.position = SCNVector3(0, 0, -1) sceneView.scene.rootNode.addChildNod e(node)
29. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO REAL WORLD (PLANE) sceneView.session.delegate = self let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.session.run(configuration) func session(_ session: ARSession, didAdd anchors: [ARAnchor]) { for case _ as ARPlaneAnchor in anchors { print("plane detected") } } @objc func tap(gesture: UITapGestureRecognizer) { let location = sceneView.center let results = sceneView.hitTest(location, types: .existingPlane) if let anchor = results.first?.anchor { let node = box() // !!!: this transform is the center of the plane node.position = SCNVector3Make(anchor.transform.columns.3.x, anchor.transform.columns.3.y, anchor.transform.columns.3.z) sceneView.scene.rootNode.addChildNode(node) } } 29 @rompelstilchen
30. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - SCENEKIT 30 @rompelstilchen let location = gesture.location(in: sceneView) let results = sceneView.hitTest(location, options: nil) if let result = results.first?.node { ... }
31. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - ARKIT 31 @rompelstilchen // find the current nodes in the field of view if let pov = sceneView.pointOfView { let nodes = sceneView.nodesInsideFrustum(of: pov) for node in nodes { ... } }
32. @rompelstilchen ENRICHED UI WITH ARKIT LIMITATIONS Horizontal Plane detection only Unstable API (beta) Hard to test (session reset, simulator) Real-world position can be jumpy 32
33. @rompelstilchen ENRICHED UI WITH ARKIT GOING FURTHER Apple HIG for AR @madewitharkit WWDC :) 33
34. @rompelstilchen THANK YOU! SLIDES AVAILABLE ON SPEAKER DECK: HTTP://BIT.LY/2JLOJWX HUGUES BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 34