Enriched UI using ARKit

This talk is a shorter version of my previous talk at SwiftConf 2017.
It have been given at SwiftParis.
It focus on the SceneKit / ARKit integration (not SpriteKit)

Transcript

1. @rompelstilchen ENRICHED UI WITH ARKIT HUGUES BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 1
2. @rompelstilchen ENRICHED UI WITH ARKIT TOPICS ARKit Overview / Primer SceneKit Primer Sample UI Demo Limitations Going further 2
3. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT OVERVIEW Inertial Measurement Unit Visual Inertial Odometry Dead reckoning 6D (3D translation/3D rotation) 3
4. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT OVERVIEW 2D Plane detection Metric scale Light estimation Integration with SpriteKit, SceneKit or Metal 4
5. @rompelstilchen 5
6. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER ARSessionConfiguration ARSession ARFrame ARAnchor 6
7. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARSESSION Manage AR processing Reset Tracking Session Updates 7
8. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARFRAME Camera image to render the background of the scene Tracking informations to find device's location, orientation and state Scene informations such as light estimate and location in space 8
9. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARANCHOR Real-world position and orientation Managed via ARSession Mainly created for you by ARKit 9
10. @rompelstilchen ENRICHED UI WITH SCENEKIT HUGUES BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 10
11. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT - PRIMER SCNView SCNScene SCNNode (SCNSprite, SCNTextNode, …) SCNGeometry, SCNMaterial 11
12. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT / ARKIT ARSCNView > SCNView + ARSession ARAnchor <> SCNNode That’s pretty much it (again) 12
13. @rompelstilchen DEMO ENRICHED UI WITH ARKIT / SCENEKIT 13
14. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT / ARKIT - STEPS Position content using ARAnchor and/or the rootNode (initial camera position) Interact with content using UIGesture or the device FOV/position Convert point back and forth ARKit do the rest 14
15. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - FIX POSITION // fix position relative to the point of view // kind of useless let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0) box.firstMaterial?.diffuse.contents = UIColor.red let node = SCNNode(geometry: box) node.position = SCNVector3(0, 0, -1) sceneView.pointOfView?.addChildNode( node) // it’s a UIView subclass let label = UILabel() view.addSubview(label) 15 @rompelstilchen
16. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO INITIAL CAMERA POSITION 16 @rompelstilchen // the root node is updated relative to camera by ARKit let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0) box.firstMaterial?.diffuse.contents = UIColor.red let node = SCNNode(geometry: box) node.position = SCNVector3(0, 0, -1) sceneView.scene.rootNode.addChildNod e(node)
17. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO REAL WORLD (PLANE) sceneView.session.delegate = self let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.session.run(configuration) func session(_ session: ARSession, didAdd anchors: [ARAnchor]) { for case _ as ARPlaneAnchor in anchors { print("plane detected") } } @objc func tap(gesture: UITapGestureRecognizer) { let location = sceneView.center let results = sceneView.hitTest(location, types: .existingPlane) if let anchor = results.first?.anchor { let node = box() // !!!: this transform is the center of the plane node.position = SCNVector3Make(anchor.transform.columns.3.x, anchor.transform.columns.3.y, anchor.transform.columns.3.z) sceneView.scene.rootNode.addChildNode(node) } } 17 @rompelstilchen
18. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - SCENEKIT 18 @rompelstilchen let location = gesture.location(in: sceneView) let results = sceneView.hitTest(location, options: nil) if let result = results.first?.node { ... }
19. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - ARKIT 19 @rompelstilchen // find the current nodes in the field of view if let pov = sceneView.pointOfView { let nodes = sceneView.nodesInsideFrustum(of: pov) for node in nodes { ... } }
20. @rompelstilchen ENRICHED UI WITH ARKIT LIMITATIONS Horizontal Plane detection only No scene reconstruction (occlusion) Hard to test (session reset, simulator) Real-world position can be jumpy No persistance (session based) 20
21. @rompelstilchen ENRICHED UI WITH ARKIT GOING FURTHER Apple HIG for AR @madewitharkit WWDC :) 21
22. @rompelstilchen THANK YOU! SLIDES AVAILABLE ON SPEAKER DECK: HTTP://BIT.LY/2JLOJWX HUGUES BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 22