Getting Started with ARKit

Presented to NSMeetup in San Francisco, December 5, 2017.

1. Getting Started with ARKit Building augmented reality apps for iOS 11 Warren Moore @ NSMeetup 5 December 2017
2. A little about me • Worked on Camera & Photo apps at Apple (2014) • Wrote a book on Metal • Worked on Metal ecosystem at Apple (2016) • Now independent again 2
3. Agenda • What is AR[Kit]? • Building an ARKit app • Augmenting your face with iPhone X • Further considerations • Q&A 3
4. What is AR[Kit]?
5. What is AR? “[Taking] digital or computer-generated information, whether it be images, audio, video, and touch or haptic sensations and overlaying them [on] a real-time environment” Kipper, Greg; Rampolla, Joseph. Augmented Reality: An Emerging Technologies Guide to AR 5
7. A brief history 7
9. What is ARKit? • Tracking • Scene understanding • Rendering 9
10. Tracking • Six degree of freedom (6 DOF) world tracking • Visual inertial odometry • Markerless, bootstrapping 10
11. Scene Understanding • Feature detection • Horizontal planes • Facial poses and expressions (iPhone X) • Hit testing • Light estimation 11
12. Rendering • Deep integration with game frameworks • SpriteKit • SceneKit • Hooks and utilities for custom rendering (Metal, etc.) 12
15. Building an ARKit app
17. Step by step • Configuring and running a session • Receiving feature point updates • Adding geometry to visualize anchors • Interacting with the scene 17
18. But first! 18 NSCameraUsageDescription NSMicrophoneUsageDescription
19. Getting an ARSession 19 let session = ARSession() let session = arView.session
20. Creating a configuration 20 let config = ARWorldTrackingConfiguration() config.planeDetection = .horizontal
21. Running a configuration 21 session.run(configuration, options: .resetTracking)
22. ARSCNViewDelegate 22 arView.delegate = self
23. ARSCNViewDelegate 23 func renderer(_ renderer: SCNSceneRenderer, didAdd node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { return } addPlaneGeometry(for: planeAnchor, node) } func renderer(_ renderer: SCNSceneRenderer, didUpdate node: SCNNode, for anchor: ARAnchor) { guard let planeAnchor = anchor as? ARPlaneAnchor else { return } guard let planeNode = node.childNodes.first else { return } updatePlaneGeometry(for: planeAnchor, planeNode) }
24. Adding geometry to an anchor 24 func addPlaneGeometry(for anchor: ARPlaneAnchor, _ node: SCNNode) { let planeGeometry = SCNPlane(width: 1, height: 1) let material = planeGeometry.firstMaterial! material.diffuse.contents = UIImage(named: "grid.png") material.diffuse.wrapS = .repeat material.diffuse.wrapT = .repeat let planeNode = SCNNode(geometry: planeGeometry) node.addChildNode(planeNode) }
25. Updating plane geometry 25 func updatePlaneGeometry(for anchor: ARPlaneAnchor, _ node: SCNNode) { node.position = SCNVector3(anchor.center.x, 0, anchor.center.z) node.transform = SCNMatrix4MakeRotation(-Float.pi / 2, 1, 0, 0) let planeGeometry = node.geometry as! SCNPlane planeGeometry.width = CGFloat(anchor.extent.x) planeGeometry.height = CGFloat(anchor.extent.z) let material = planeGeometry.firstMaterial! material.diffuse.contentsTransform = SCNMatrix4MakeScale(anchor.extent.x, anchor.extent.z, 1) }
26. Hit testing 26 override func touchesMoved(_ touches: Set<UITouch>, with event: UIEvent?) { super.touchesMoved(touches, with: event) if let firstTouch = touches.first { let point = firstTouch.location(in: arView) addInk(at: point) } }
27. Going splat 27 func addInk(at point: CGPoint) { let results = arView.hitTest(point, types: .existingPlaneUsingExtent) if let nearestResult = results.first { guard let anchor = nearestResult.anchor as? ARPlaneAnchor else { return } guard let planeNode = arView.node(for: anchor) else { return } let splatGeometry = SCNPlane(width: 0.25, height: 0.25) let material = splatGeometry.firstMaterial! material.diffuse.contents = UIImage(named: "splat-pink") let splatNode = SCNNode(geometry: splatGeometry) let planeOrienation = SCNMatrix4MakeRotation(-Float.pi / 2, 1, 0, 0) splatNode.transform = SCNMatrix4Mult(planeOrienation, SCNMatrix4(nearestResult.localTransform)) planeNode.addChildNode(splatNode) } }
29. Augmenting your face
30. ARFaceTrackingConfiguration • Uses TrueDepth camera exclusive to iPhone X • Higher-fidelity light estimation 30
31. ARFaceAnchor • 6 DOF facial pose • Fine-grained facial expression via blend shapes • Face mesh - ARFaceGeometry, ARSCNFaceGeometry 31
32. Visualizing blend shapes
33. Blend shape locations 33 browDownLeft browDownRight browInnerUp browOuterUpLeft browOuterUpRight cheekPuff cheekSquintLeft cheekSquintRight eyeBlinkLeft eyeBlinkRight eyeLookDownLeft eyeLookDownRight eyeLookInLeft eyeLookInRight eyeLookOutLeft eyeLookOutRight eyeLookUpLeft eyeLookUpRight eyeSquintLeft eyeSquintRight eyeWideLeft eyeWideRight jawForward jawLeft jawOpen jawRight mouthClose mouthDimpleLeft mouthDimpleRight mouthFrownLeft mouthFrownRight mouthFunnel mouthLeft mouthLowerDownLeft mouthLowerDownRight mouthPressLeft mouthPressRight mouthPucker mouthRight mouthRollLower mouthRollUpper mouthShrugLower mouthShrugUpper mouthSmileLeft mouthSmileRight mouthStretchLeft mouthStretchRight mouthUpperUpLeft mouthUpperUpRight noseSneerLeft noseSneerRight
35. Step by step • Configuring and running a face-tracking session • Receiving face anchor updates • Adding/updating geometry according to blend shapes 35
36. Face Tracking Configuration 36 let config = ARFaceTrackingConfiguration() config.isLightEstimationEnabled = true
37. Loading the Model 37 // Load model file from bundle let nutcrackerNode = SCNReferenceNode(url: url)! nutcrackerNode.load() // Store nodes we will be animating mouthNode = nutcrackerNode.childNode(withName: "mouth", recursively: true) eyebrowLeftNode = nutcrackerNode.childNode(withName: "left_eyebrow", recursively: true) eyebrowRightNode = nutcrackerNode.childNode(withName: "right_eyebrow", recursively: true) // Store initial positions of animated nodes so we can offset them mouthPosition = mouthNode.position eyebrowLeftPosition = eyebrowLeftNode.position eyebrowRightPosition = eyebrowRightNode.position
38. Animating by blend shapes 38 func updateModel(_ faceAnchor: ARFaceAnchor) { let blendShapes = faceAnchor.blendShapes guard let browUp = blendShapes[.browInnerUp] as? Float else { return } guard let jawOpen = blendShapes[.jawOpen] as? Float else { return } eyebrowLeftNode.position.y = eyebrowLeftPosition.y + (browUp * browRaiseHeight) eyebrowRightNode.position.y = eyebrowRightPosition.y + (browUp * browRaiseHeight) mouthNode.position.y = mouthPosition.y - (jawOpen * jawHeight) }
40. Further considerations
41. • Finding AR content • AR “aesthetic” • Accessibility 41 Topics
42. Where to find AR content • Not an artist? Not a problem! • Multiple high-quality content markets - Sketchfab - Google Poly - CGTrader, Turbosquid, etc. • Easily convert to OBJ/DAE/SCN 42
45. A E S T H E T I C
46. Accessibility • As with all custom drawing (Core Graphics, Metal, etc.), it’s not automatic • Consider adopting the UIAccessibilityContainer protocol • See Sommer Panage’s talk at Realm https://academy.realm.io/posts/sommer-panage-accessibility-implementation-ios/ • Matt Gemmell has some timeless words on this too https://mattgemmell.com/accessibility-for-iphone-and-ipad-apps/ 46
47. Accessibility redux • By definition, you’re already augmenting the world • Look for opportunities to aid users with visual, motor, or cognitive challenges • Machine learning can help everyone see better - Object identification, navigation, situational awareness 47
48. Resources / Q&A • WWDC 2017 and Fall 2017 Videos https://developer.apple.com/arkit/ • Mohammad Azam’s book and course on Udemy https://gumroad.com/l/QOIq https://www.udemy.com/mastering-arkit-for-ios-using-swift • Upload ARKit weekend course (Dec. 8-10) https://upload.io/courses/ux-of-ar-design-arkit/ 48
49. Thank you!