Getting Started with ARKit Building augmented reality apps for iOS 11 Warren Moore @ NSMeetup 5 December 2017 

A little about me • Worked on Camera & Photo apps at Apple (2014) • Wrote a book on Metal • Worked on Metal ecosystem at Apple (2016) • Now independent again 2 

Agenda • What is AR[Kit]? • Building an ARKit app • Augmenting your face with iPhone X • Further considerations • Q&A 3 

What is AR[Kit]? 

What is AR? “[Taking] digital or computer-generated information, whether it be images, audio, video, and touch or haptic sensations and overlaying them [on] a real-time environment” Kipper, Greg; Rampolla, Joseph. Augmented Reality: An Emerging Technologies Guide to AR 5 

No content

A brief history 7 

No content

What is ARKit? • Tracking • Scene understanding • Rendering 9 

Tracking • Six degree of freedom (6 DOF) world tracking • Visual inertial odometry • Markerless, bootstrapping 10 

Scene Understanding • Feature detection • Horizontal planes • Facial poses and expressions (iPhone X) • Hit testing • Light estimation 11 

Rendering • Deep integration with game frameworks • SpriteKit • SceneKit • Hooks and utilities for custom rendering (Metal, etc.) 12 

13 

14 

Building an ARKit app 

No content

Step by step • Configuring and running a session • Receiving feature point updates • Adding geometry to visualize anchors • Interacting with the scene 17 

But first! 18 NSCameraUsageDescription NSMicrophoneUsageDescription 

Getting an ARSession 19 let session = ARSession() let session = arView.session 

Creating a configuration 20 let config = ARWorldTrackingConfiguration() config.planeDetection = .horizontal 

Running a configuration 21 session.run(configuration, options: .resetTracking) 

ARSCNViewDelegate 22 arView.delegate = self 

ARSCNViewDelegate 23 func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { return } addPlaneGeometry(for: planeAnchor, node) } func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { return } guard let planeNode = node.childNodes.first else { return } updatePlaneGeometry(for: planeAnchor, planeNode) } 

Adding geometry to an anchor 24 func addPlaneGeometry(for anchor: ARPlaneAnchor, _ node: SCNNode) { let planeGeometry = SCNPlane(width: 1, height: 1) let material = planeGeometry.firstMaterial! material.diffuse.contents = UIImage(named: "grid.png") material.diffuse.wrapS = .repeat material.diffuse.wrapT = .repeat let planeNode = SCNNode(geometry: planeGeometry) node.addChildNode(planeNode) } 

Updating plane geometry 25 func updatePlaneGeometry(for anchor: ARPlaneAnchor, _ node: SCNNode) { node.position = SCNVector3(anchor.center.x, 0, anchor.center.z) node.transform = SCNMatrix4MakeRotation(-Float.pi / 2, 1, 0, 0) let planeGeometry = node.geometry as! SCNPlane planeGeometry.width = CGFloat(anchor.extent.x) planeGeometry.height = CGFloat(anchor.extent.z) let material = planeGeometry.firstMaterial! material.diffuse.contentsTransform = SCNMatrix4MakeScale(anchor.extent.x, anchor.extent.z, 1) } 

Hit testing 26 override func touchesMoved(_ touches: Set, with event: UIEvent?) { super.touchesMoved(touches, with: event) if let firstTouch = touches.first { let point = firstTouch.location(in: arView) addInk(at: point) } } 

Going splat 27 func addInk(at point: CGPoint) { let results = arView.hitTest(point, types: .existingPlaneUsingExtent) if let nearestResult = results.first { guard let anchor = nearestResult.anchor as? ARPlaneAnchor else { return } guard let planeNode = arView.node(for: anchor) else { return } let splatGeometry = SCNPlane(width: 0.25, height: 0.25) let material = splatGeometry.firstMaterial! material.diffuse.contents = UIImage(named: "splat-pink") let splatNode = SCNNode(geometry: splatGeometry) let planeOrienation = SCNMatrix4MakeRotation(-Float.pi / 2, 1, 0, 0) splatNode.transform = SCNMatrix4Mult(planeOrienation, SCNMatrix4(nearestResult.localTransform)) planeNode.addChildNode(splatNode) } } 

No content

Augmenting your face 

ARFaceTrackingConfiguration • Uses TrueDepth camera exclusive to iPhone X • Higher-fidelity light estimation 30 

ARFaceAnchor • 6 DOF facial pose • Fine-grained facial expression via blend shapes • Face mesh - ARFaceGeometry, ARSCNFaceGeometry 31 

Visualizing blend shapes 

Blend shape locations 33 browDownLeft browDownRight browInnerUp browOuterUpLeft browOuterUpRight cheekPuff cheekSquintLeft cheekSquintRight eyeBlinkLeft eyeBlinkRight eyeLookDownLeft eyeLookDownRight eyeLookInLeft eyeLookInRight eyeLookOutLeft eyeLookOutRight eyeLookUpLeft eyeLookUpRight eyeSquintLeft eyeSquintRight eyeWideLeft eyeWideRight jawForward jawLeft jawOpen jawRight mouthClose mouthDimpleLeft mouthDimpleRight mouthFrownLeft mouthFrownRight mouthFunnel mouthLeft mouthLowerDownLeft mouthLowerDownRight mouthPressLeft mouthPressRight mouthPucker mouthRight mouthRollLower mouthRollUpper mouthShrugLower mouthShrugUpper mouthSmileLeft mouthSmileRight mouthStretchLeft mouthStretchRight mouthUpperUpLeft mouthUpperUpRight noseSneerLeft noseSneerRight 

No content

Step by step • Configuring and running a face-tracking session • Receiving face anchor updates • Adding/updating geometry according to blend shapes 35 

Face Tracking Configuration 36 let config = ARFaceTrackingConfiguration() config.isLightEstimationEnabled = true 

Loading the Model 37 // Load model file from bundle let nutcrackerNode = SCNReferenceNode(url: url)! nutcrackerNode.load() // Store nodes we will be animating mouthNode = nutcrackerNode.childNode(withName: "mouth", recursively: true) eyebrowLeftNode = nutcrackerNode.childNode(withName: "left_eyebrow", recursively: true) eyebrowRightNode = nutcrackerNode.childNode(withName: "right_eyebrow", recursively: true) // Store initial positions of animated nodes so we can offset them mouthPosition = mouthNode.position eyebrowLeftPosition = eyebrowLeftNode.position eyebrowRightPosition = eyebrowRightNode.position 

Animating by blend shapes 38 func updateModel(_ faceAnchor: ARFaceAnchor) { let blendShapes = faceAnchor.blendShapes guard let browUp = blendShapes[.browInnerUp] as? Float else { return } guard let jawOpen = blendShapes[.jawOpen] as? Float else { return } eyebrowLeftNode.position.y = eyebrowLeftPosition.y + (browUp * browRaiseHeight) eyebrowRightNode.position.y = eyebrowRightPosition.y + (browUp * browRaiseHeight) mouthNode.position.y = mouthPosition.y - (jawOpen * jawHeight) } 

No content

Further considerations 

• Finding AR content • AR “aesthetic” • Accessibility 41 Topics 

Where to find AR content • Not an artist? Not a problem! • Multiple high-quality content markets - Sketchfab - Google Poly - CGTrader, Turbosquid, etc. • Easily convert to OBJ/DAE/SCN 42 

No content

No content

A E S T H E T I C 

Accessibility • As with all custom drawing (Core Graphics, Metal, etc.), it’s not automatic • Consider adopting the UIAccessibilityContainer protocol • See Sommer Panage’s talk at Realm https://academy.realm.io/posts/sommer-panage-accessibility-implementation-ios/ • Matt Gemmell has some timeless words on this too https://mattgemmell.com/accessibility-for-iphone-and-ipad-apps/ 46 

Accessibility redux • By definition, you’re already augmenting the world • Look for opportunities to aid users with visual, motor, or cognitive challenges • Machine learning can help everyone see better - Object identification, navigation, situational awareness 47 

Resources / Q&A • WWDC 2017 and Fall 2017 Videos https://developer.apple.com/arkit/ • Mohammad Azam’s book and course on Udemy https://gumroad.com/l/QOIq https://www.udemy.com/mastering-arkit-for-ios-using-swift • Upload ARKit weekend course (Dec. 8-10) https://upload.io/courses/ux-of-ar-design-arkit/ 48 

Thank you!