Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Enriched UI using ARKit

Enriched UI using ARKit

Slides of my talk at SwiftConf 2017 in Cologne. Video available at http://bit.ly/2eRnUvF

Hugues Bernet-Rollande

September 07, 2017
Tweet

More Decks by Hugues Bernet-Rollande

Other Decks in Programming

Transcript

  1. @rompelstilchen ENRICHED UI WITH ARKIT TOPICS ARKit Overview / Primer

    SpriteKit / SceneKit Primer Sample UI Demo Limitations Going further 2
  2. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT OVERVIEW Visual Inertial Odometry

    Inertial Measurement Unit Dead reckoning 6D (3D translation/3D rotation) 3
  3. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT OVERVIEW 2D Plane detection

    Metric scale Light estimation Integration with SpriteKit, SceneKit, Metal or custom 4
  4. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARSESSION Manage

    AR processing Reset Tracking Session Updates 7
  5. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARFRAME Camera

    image to render the background of the scene Tracking informations to find device's location, orientation and state Scene informations such as light estimate and location in space 8
  6. @rompelstilchen ENRICHED UI WITH ARKIT ARKIT PRIMER - ARANCHOR Real-world

    position and orientation Managed via ARSession Also created for you by ARKit 9
  7. @rompelstilchen ENRICHED UI WITH ARKIT SPRITEKIT - PRIMER SKView SKScene

    SKNode (SKSprite, SKTextNode, …) (SKAction, SKTransition, SKTexture, ..) 12
  8. @rompelstilchen ENRICHED UI WITH ARKIT SPRITEKIT / ARKIT ARSKView >

    SKView + ARSession ARAnchor <> SKNode That’s pretty much it! 13
  9. @rompelstilchen ENRICHED UI WITH ARKIT SPRITEKIT / ARKIT - STEPS

    Position content using ARAnchor Interact with content using UIGesture, UIResponder or camera position tracking Convert point back and forth ARKit do the rest 15
  10. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - FIX SCREEN

    POSITION // Fix content in the center of the screen let node = SKShapeNode(rectOf: CGSize(width: 20, height: 20), cornerRadius: 10) node.fillColor = .blue sceneView.scene?.addChild(node) // it’s a UIView subclass let label = UILabel() sceneView.addSubview(label) 16 @rompelstilchen
  11. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO

    CAMERA CURRENT POSITION if let camera = session.currentFrame?.camera { var translation = matrix_identity_float4x4 translation.columns.3.z = -1 let transform = simd_mul(camera.transform, translation) let anchor = ARAnchor(transform: transform) session.add(anchor: anchor) } func view(_ view: ARSKView, nodeFor anchor: ARAnchor) -> SKNode? { return SKLabelNode(text: “Hello World") } 17 @rompelstilchen
  12. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO

    REAL WORLD (PLANE) 18 @rompelstilchen // on touch let location = gesture.location(in: sceneView) // plane detection enabled and detected let results = sceneView.hitTest(location, types: .existingPlane) if let anchor = results.first?.anchor { sceneView.session.add(anchor: anchor) } func view(_ view: ARSKView, nodeFor anchor: ARAnchor) -> SKNode? { return SKLabelNode(text: “Hello World") }
  13. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - UIKIT/SPRITEKIT

    class SKTouch: SKShapeNode { override init() { super.init() isUserInteractionEnabled = true } override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) { … } } 19 @rompelstilchen
  14. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - UIKIT/SPRITEKIT

    20 @rompelstilchen @objc func tap_(gesture: UITapGestureRecognizer) { guard gesture.state == .ended else { return } let location = gesture.location(in: sceneView) let nodes = sceneView.scene?.nodes(at: location) if let node = nodes?.first { … } }
  15. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - DEVICE

    21 @rompelstilchen func session(_ session: ARSession, didUpdate frame: ARFrame) { let location = sceneView.center if let sceneLocation = sceneView.scene?.convertPoint(fromView: location), let nodes = sceneView.scene?.nodes(at: sceneLocation) { if let node = nodes.first { ... } } }
  16. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT - PRIMER SCNView SCNScene

    SCNNode (SCNSprite, SCNTextNode, …) SCNGeometry, SCNMaterial 23
  17. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT / ARKIT ARSCNView >

    SCNView + ARSession ARAnchor <> SCNNode That’s pretty much it (again) 24
  18. @rompelstilchen ENRICHED UI WITH ARKIT SCENEKIT / ARKIT - STEPS

    Position content using ARAnchor and/or the rootNode (initial camera position) Interact with content using UIGesture or the device FOV/position Convert point back and forth ARKit do the rest 26
  19. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - FIX POSITION

    // fix position relative to the point of view // kind of useless let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0) box.firstMaterial?.diffuse.contents = UIColor.red let node = SCNNode(geometry: box) node.position = SCNVector3(0, 0, -1) sceneView.pointOfView?.addChildNode( node) // it’s a UIView subclass let label = UILabel() view.addSubview(label) 27 @rompelstilchen
  20. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO

    INITIAL CAMERA POSITION 28 @rompelstilchen // the root node is updated relative to camera by ARKit let box = SCNBox(width: 0.1, height: 0.1, length: 0.1, chamferRadius: 0) box.firstMaterial?.diffuse.contents = UIColor.red let node = SCNNode(geometry: box) node.position = SCNVector3(0, 0, -1) sceneView.scene.rootNode.addChildNod e(node)
  21. @rompelstilchen ENRICHED UI WITH ARKIT POSITION CONTENT - RELATIVE TO

    REAL WORLD (PLANE) sceneView.session.delegate = self let configuration = ARWorldTrackingConfiguration() configuration.planeDetection = .horizontal sceneView.session.run(configuration) func session(_ session: ARSession, didAdd anchors: [ARAnchor]) { for case _ as ARPlaneAnchor in anchors { print("plane detected") } } @objc func tap(gesture: UITapGestureRecognizer) { let location = sceneView.center let results = sceneView.hitTest(location, types: .existingPlane) if let anchor = results.first?.anchor { let node = box() // !!!: this transform is the center of the plane node.position = SCNVector3Make(anchor.transform.columns.3.x, anchor.transform.columns.3.y, anchor.transform.columns.3.z) sceneView.scene.rootNode.addChildNode(node) } } 29 @rompelstilchen
  22. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - SCENEKIT

    30 @rompelstilchen let location = gesture.location(in: sceneView) let results = sceneView.hitTest(location, options: nil) if let result = results.first?.node { ... }
  23. @rompelstilchen ENRICHED UI WITH ARKIT INTERACT WITH CONTENT - ARKIT

    31 @rompelstilchen // find the current nodes in the field of view if let pov = sceneView.pointOfView { let nodes = sceneView.nodesInsideFrustum(of: pov) for node in nodes { ... } }
  24. @rompelstilchen ENRICHED UI WITH ARKIT LIMITATIONS Horizontal Plane detection only

    Unstable API (beta) Hard to test (session reset, simulator) Real-world position can be jumpy 32
  25. @rompelstilchen THANK YOU! SLIDES AVAILABLE ON SPEAKER DECK: HTTP://BIT.LY/2JLOJWX HUGUES

    BERNET-ROLLANDE @ROMPELSTILCHEN GITHUB.COM/HUGUESBR FREELANCE IOS DEVELOPER 34